Analytic Services

Question 1

#Glue
What are Glue Worker Types?

Answer 1

AWS Glue comes with 3 worker types

Standard - 4 vCPU, 16GB RAM, 50GB, 2 Spark executors ⇒ 1 DPU

G.1X - 4 vCPU, 16GB RAM, 64GB, 1 Spark executors ⇒ 1 DPU

G.2X - 8 vCPU, 32GB RAM, 128GB, 1 Spark executors ⇒ 2 DPU

1 DPU can run 8 Spark executors

G1.X for jobs that are memory intensive
G2.X for jobs that uses AWS Glue ML workloads such as ML Transforms

Question 2

#EMR
What are three different node types in a EMR cluster?

Answer 2

• Master or Leader nodes
  
  • manage the cluster
  • Single EC2 instance
• Core nodes: host HDFS data and run tasks
• Task nodes: run tasks not hosting data
  
  • No risk of data loss when removing them
  • Good use of spot instance

Question 3

#EMR
What is EMRFS Consistent View?

Answer 3

EMRFS Consistent View is an optional feature that allows EMR cluster to check for list and read-after-write consistency for S3 objects written or synced with EMRFS.

Why? S3’s eventual consistency

Question 4

#EMR 
How does EMRFS Consistent View works?

Answer 4

EMRFS Consistent View uses an Amazon DynamoDB to store object metadata and track consistency with S3 (EMRFS Metadata Store)

• DynamoDB by default has 400 read capacity and 100 write capacity - you need to configure DDB according to # of objects being tracked
• You can configure # of retries and retry period (# seconds) - consistent view will retry

Question 5

#EMR
What are the storage options for a EMR cluster?

Can you add or detach a EBS volumes to a running EMR cluster?

Answer 5

• HDFS - distributed local storage
• Ephemeral
• Default block size of 128 MB
• EMRFS: access S3 as if it were HDFS
• EMRFS Consistent View - optional for S3 consistency
• Use DynamoDB to track consistency - pay attention to capacity limits
• Why? S3 eventual consistency
• Local file system
• EBS for HDFS
• Will be deleted if the cluster gets terminated
• Can not add or detach EBS volumes to a running cluster, only at creation time

Question 6

#EMR
What does the Spark stack consist of?

Answer 6

• Spark SQL (structure); Spark Streaming (real-time); MLLib; GraphX (graph processing)
• Spark Core
• Standalone Scheduler; YARN; Mesos

Question 7

#EMR
What is Hive?

Answer 7

Hive - data warehouse and analytic infrastructure built on top of Hadoop

• Hive uses SQL like language called Hive SQL

Question 8

#EMR
What is Tez?

Answer 8

Tez is an extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop;

Both Tez and MapReduce are execution engine in Hive

Question 9

#EMR
What is Presto?

Answer 9

Pesto is an open source in-memory distributed fast SQL query engine designed for interactive queries against PB of data from different sources

• Run interactive analytics queries against a variety of data sources with size ranging from GB to PB at once and query across them
• Interactive queries at PB scale - faster than Hive
• Optimized for OLAP - analytics queries, data warehousing
• This is what Athena use under the hood
  Expose JDBC, CLI, and Tableau interfaces

Question 10

#EMR
What are EMR Notebooks?

Answer 10

EMR Notebooks are similar Zeppelin but with more AWS Integration

• Notebooks backup to S3
• Provision clusters from the notebook
• Host inside a VPC
• Access ONLY via AWS Console

Question 11

#EMR
What is HUE?

Answer 11

HUE stands for Hadoop User Experience. It is a Open source web interface for Apache Hadoop and other non-Hadoop applications running on EMR

• You can browse S3 / HDFS / Hbase / Zookeeper
• You can move data between S3 and HDFS

Question 12

#EMR
What is Flume?

Answer 12

Flume is another way to streaming data (e.g. log data) into your EMR cluster.

• Build-in sinks for HDFS and HBase
• Source ==> Channels ==> Sink ==> Target

Question 13

#EMR
What is MXNet?

Answer 13

MXNet is an alternative to Tensorflow for building neural networks. MXNet is included in EMR.

Question 14

#EMR
What is S3DistCP?

Answer 14

S3DistCP is a tool for copying large amount of data between S3 and HDFS

• Uses MapReduce to copy in a distributed manner
• Suitable for parallel copying of large # of objects across buckets and accounts

Question 15

#EMR
What are the Hive integration with AWS?

Answer 15

• S3: load data / script / partition information

- DynamoDB as an external table. Hive can process and join data stored in DynamoDB

Question 16

#Quicksight
What is a KPI Chart?

Answer 16

KPI Charts use a key performance indicator (KPI) to visualize a comparison between a key value and its target value.

Question 17

#DynamoDB
What is WCU and RCU for DynamoDB?

Answer 17

1 WCU = 1KB/s WRITE

1 RCU = 2 eventual consistent READ of 4KB/s; 1 consistent READ of 4KB/s

Question 18

#S3
What is Glacier Select?

Answer 18

Glacier Select allow you query Glacier data with simple SQL queries and get results in minutes, without need to restore to S3.

Question 19

#IoT
What are types of identity principals for device or client authentication supported by AWS IoT?

Answer 19

1. X.509 cert
2. IAM users, groups, and roles
3. Amazon Cognito identities

I think you can also implement Federated Identity via Cognito since it can use

Question 20

#Redshift
What is Redshift's Elastic Resize?

Answer 20

Redshift’s Elastic Resize allow you add / remove nodes and also change node types.

However, Elastic resize only holds connections open if you only change the number of nodes, not the node type.

If you want to minimizes the downtime involved, you might still use the snapshot / restore / resize approach with classic resize

https://aws.amazon.com/blogs/big-data/scale-your-amazon-redshift-clusters-up-and-down-in-minutes-to-get-the-performance-you-need-when-you-need-it/

Question 21

#EMR
Which compression algorithm are splittable?

Answer 21

BZIP and LZO are splittable, great for parallel processing

GZIP and SNAPPY are NOT splittable

Question 22

#EMR
What is HBase Read-replica in S3?

Answer 22

Amazon EMR version 5.7.0+ allows you to maintain read-only copies of data in Amazon S3.

You can access the data from the read-replica cluster to perform read operations simultaneously, and in the event that the primary cluster becomes unavailable.

Question 23

#EMR
What are the 3 ways that HBase can integrate with S3

Answer 23

1. HBase on S3 (S3 storage mode) - Storage of HBase StoreFiles and metadata in S3
2. Snapshot of HBase to S3 - this is for the HBase that do not use S3
3. HBase Read-replcas in S3

Question 24

#EMR
What is Ganglia?

Answer 24

Ganglia is a a scalable, distributed system designed to monitor clusters and grids while minimizing the impact on their performance.

Ganglia is installed on the Master Node and Ganglia is the operational dashboard provided with EMR.

Question 25

``` #EMR What is Apache Sqoop? ```

Answer 25

Sqoop is an open-source system for transferring data between Hadoop and relational databases (in parallel). You can use Sqoop to copy data from RDBMS to EMR for analysis

Question 26

``` #QuickSight What does HeatMap chart used for? ```

Answer 26

Heatmap is used to highlight outliers and trends using color.

Question 27

``` #QuickSight What does Scatter Plot used for? What does Tree Map used for? ```

Answer 27

Scatter Plot is used for visualizing two or three measures for a dimension. Tree Map is used for visualizing one or two measures for a dimension, use tree maps.

Question 28

``` #EMR What are the advantages of HBase on S3? ```

Answer 28

1. The HBase root directory is stored in Amazon S3, including HBase store files and table metadata. The data is persist outside cluster and available across AZs, no snapshot / recover is needed 2. Separate compute and storage, so you can scale EMR cluster for your compute requirements 3. You can have a read-replica cluster and perform read operation concurrently, even when the primary cluster is not available

Question 29

``` #EMR What is HCatalog? ```

Answer 29

HCatalog is a tool that allows you to access Hive metastore tables within Pig, Spark SQL, and/or custom MapReduce applications.

Question 30

``` #Redshift Does Redshift support multi-AZ deployment? ```

Answer 30

No. Currently, Amazon Redshift only supports Single-AZ deployments. - You can create Redshift clusters in each AZ and stream / COPY same data into them; OR - You can use Redshift Spectrum to query the same S3 data with Redshift clusters in different AZ

Question 31

``` #QuickSight What is Pivotal Table used for? ```

Answer 31

You can use Pivotal table to interactively explore multi-dimensional data, applying statistical functions to different rows and columns and sorting them in different ways.

Question 32

``` #EMR Why use external metastore for Hive? What are the choices for Hive external metastore? ```

Answer 32

External metastore is for persisting metastore beyond the cluster’s life span. By default, Hive metastore is in a MySQL on master node. But you can choose use external metastore 1. AWS Glue Data Catalog 2. Amazon RDS or Aurora

Question 33

``` #EMR #S3 What is S3 Select? How does S3 Select work with EMR and Hive? ```

Answer 33

S3 Select allows applications to retrieve only a subset of data from an S3 object. For EMR, S3 Select can improve the performance of data processing by reduced the data movement between S3 and EMR This is also called S3 Select Push Down Presto on EMR can use S3 Select Push Diwn

Question 34

``` #EMR What is Hudi? Why we need it? ```

Answer 34

Hudi open-source data management framework used to simplify incremental data processing and data pipeline development by providing record-level insert, update, upsert, and delete capabilities. - Hudi is integrated with Spark, Hive, Presto, and EMR. - With Hudi, you can apply changes to a dataset over time.

Question 35

``` #EMR What are the two Hudi dataset types? ```

Answer 35

1. Copy on Write (CoW) - default. - Data is stored in a columnar format (Parquet), and each update creates a new version of files during a write; - Better suited for read-heavy workloads on data that changes less frequently. 2. Merge on Read (MoR) - Data is stored using a combination of columnar (Parquet) and row-based (Avro) formats. - Updates are logged to row-based delta files and are compacted as needed to create new versions of the columnar files - Better suited for write- or change-heavy workloads with fewer reads.

Question 36

``` #EMR What are the 3 logical view of Hudi datasets? ```

Answer 36

1. Read Optimized View - default. latest for CoW and latest compacted data for MoR. Data may not be the latest for MOR 2. Incremental View - changes stream between actions out of CoW dataset to feed downstream jobs and ETL workflows 3. Real-time View - latest data from MoR

Question 37

``` #EMR What are the 2 options you have using Jupyter notebooks on EMR? ```

Answer 37

1. EMR Notebook - have to use it in the EMR console; stored in S3; can share among clusters; a cluster can have multiple notebooks 2. JupyterHub - it is a container running on EMR master node that can be used by multiple users; you can access via UI or CLI; stored on file system on Master node, BUT you can configure it to persist to S3

Question 38

``` #EMR What is Livy? ```

Answer 38

Livy which is included in EMR is used to talk to the Spark on EMR. You can submit job or script and manage Spark Context. JupyterHub container uses Livy.

Question 39

``` #EMR What is Mahout? ```

Answer 39

Mahout is a machine learning framework / libraries / tools for Hadoop and Spark. Mahout uses Hadoop to distribute compute across the cluster for clustering, classification, recommendation etc.

Question 40

``` #EMR What is MXNet? ```

Answer 40

Apache MXNet is an acceleration library designed for building neural networks and other deep learning applications. MXNet is included in EMR

Question 41

``` #EMR What is Oozie? ```

Answer 41

Oozie is a Hadoop job schduler and included in EMR. With EMR, you need to access Oozie via Hiue Oozie application since native interface is not supported in EMR Oozie by default persist user information and query history to local MySQL on master node, but you can configure it to presist to S3 and RDS

Question 42

``` #EMR What is Phoenix? ```

Answer 42

Apache Phoenix is used for OLTP and operational analysis with SQL against HBase store. You can connect to Phoenix using JDBC client or think client to the Phoenix Query Server running on Master node

Question 43

``` #EMR What is Pig? ```

Answer 43

Apache Pig is an open-source library that runs on top of Hadoop for data processing - Pig has SQL-like commands written in a language called Pig Latin; - Pig converts those commands into Tez jobs based on DAGs or MapReduce programs. - you can run Pig commands interactively or in batch as EMR cluster steps (from S3) - You can configure to let Pig write output to HCatalog

Question 44

``` #EMR When to use S3 Select Push Down? ```

Answer 44

- if your query filter out 50%+ orginal data - if your query predicates are supported by S3 Select, note timestamp is NOT supported - if your S3 object is in CSV format, could be optionally compressed with gzip and bzip2 - S3 SSE-C and Client side encryption is NOT supported S3 Select is not replacing using columnar (ORC or Parquet) or compressed files

Question 45

``` #EMR What is Presto Graceful Auto Scaling ```

Answer 45

EMR allows you to set a grace period for Presto tasks to keep running before the node terminates because of a scale-in resize action or an automatic scaling policy request.

Question 46

``` #EMR Can Spark on EMR work with Amazon SageMaker? ```

Answer 46

Yes. SageMaker Spark SDK is included with Spark on EMR and can be used to create ML pipelines with SageMaker Steps.

Question 47

``` #EMR What is Sqoop? ```

Answer 47

Apache Sqoop is a is a tool for transferring data between Amazon S3, Hadoop, HDFS, and RDBMS databases. Sqoop supports HCatalog and JDBC connection, and is included in EMR

Question 48

``` #EMR #DynamoDB What operations you can do on DynamoDB from a EMR cluster? ```

Answer 48

You can use EMR with Hive to load (DDB --> HDFS), query (DDB), join (DDB), import (S3 --> DDB), export (DDB --> S3)