Week 10: Big Data Framework

Question 1

What is scaling up?

Answer 1

• Using more powerful processors and more memory
• Data architecture does not significantly change

Question 2

Is scaling up a short or long term fix?

Answer 2

Short term fix

Question 3

• What is scaling out?

Answer 3

Adding servers for parallel computing

Using lots of small machines in a cluster

Question 4

Is scaling out a short or long term fix?

Answer 4

Long term as more servers may be added when needed

Question 5

When is scaling up a good option?

Answer 5

• strong internal cross-references
• need for transactional integrity

Question 6

Name a batch only framework

Answer 6

Apache Hadoop

Question 7

Name 2 stream only frameworks

Answer 7

• Apache Storm
• Apache Samza

Question 8

Name 2 hybrid frameworks

Answer 8

• Apache Spark
• Apache Flink

Question 9

Datasets in batch processing are typically:

Answer 9

• bounded: finite collection of data
• persistent: data is backed by permanent storage
• large: only option on large datasets

Question 10

Batch processing is well suited for

Answer 10

• calculations where access to all data is required (eg: averages / totals)
• tasks that require large volumes of data

Question 11

Batch processing is not appropriate when?

Answer 11

processing time is imporant

Question 12

Batch processing involves

Answer 12

• operating over a large, static dataset
• returning results at a later time
• once computation is complete

Question 13

Stream processing systems

Answer 13

• compute over data as it enters the system

Question 14

datasets in stream processing are considered;

Answer 14

unbounded

Question 15

in stream processing what does the total dataset refer to

Answer 15

the total amount of data that has entered the system so far

Question 16

Hybrid frameworks attempt to offer

Answer 16

a general solution for data processing

Question 17

For Apache Hadoop describe the following:

1. Execution Model
2. Latency
3. Programming Language
4. Fault Tolerance

Answer 17

1. Batch processing using disk storage
2. High latency
3. Java
4. Replication

Question 18

For Apache Spark describe the following:

1. Execution Model
2. Latency
3. Programming Language

Answer 18

1. batch and stream processing using memory or disk storage
2. low latency for small micro-batch size
3. Scala, Python,Java,R

Question 19

Apache spark benefits of in-memory processing

Answer 19

Runs up to 100x faster in memory

Runs up to 10x faster when it uses disk over traditional map-reduce

Question 20

Disk sharing is slow in MapReduce due to

1. 2. 3.

Answer 20

1. replication
2. serialization
3. disk IO

Question 21

Ways to create RDDs

1. 2.

Answer 21

1. parallelizing an existing collection in your driver program
2. referencing a dataset in an external storage solution (HDFS)