E5

Question 1

How distributed file systems work?

Answer 1

1. All files are broken into blocks
2. These blocks are replicated among the HDFS servers (datanodes)
3. The namenode provides a lookup service for clients accessing data and ensures the nodes are correctly replicated across the cluster

Question 2

Hadoop MapReduce

Answer 2

• Hadoop MapReduce is a distributed computing paradigm originally pioneered by Google
• Used to process data in the batch layer

Question 3

Map

Answer 3

input data is split into discrete chunks to be processed

split-apply

Question 4

Reduce

Answer 4

output of the map phase is aggregated to produce the desired result

(combine)

Question 5

The simple nature of the programming model (MapReduce) lends itself to

Answer 5

efficient and large-scale implementations across thousands of cheap nodes

Question 6

Key benefits of MapReduce

Answer 6

1. Simplicity
2. Scalability
3. Speed
4. Recovery
5. Minimal data motion

Question 7

Limitations of MapReduce

Answer 7

1. MapReduce is designed specifically for batch processing

2. Low level framework (hard to use)