Week 11 - MapReduce

Question 1

What are the steps involved in MapReduce

Answer 1

Input
Split
Mapping
Shuffling
Reducing
Output

Question 2

What is Hive

Answer 2

Hive (HQL) is like SQL, its tables are stored on HDFS as flat files, query language similar to SQL
Developed by Facebook and open source. Provides necessary SQL abstraction to integrate SQL like queries into java.

Question 3

What is Pig

Answer 3

Pig Latin is a bit like Perl, scripts written in pig latin a dataflow language, developed by yahoo, can exectute Hadoop jobs in MapReduce, Tez or Spark. Can be extended with user defined functions which can be written in a variety of languages.

Question 4

What are the differences between Hive and Pig?

Answer 4

Hive

Used by analysts
Used for reporting
Declarative SQLish language
Works on the server side of a cluster
for structured data

Pig

Used by programmers and researchers
Used for programming
Procedural data-flow language
Works on the client side of a cluster
for semi-structured data

Question 5

What are the MapReduce Architecture Components

Answer 5

• Job client
• Job tracker: monitors resources and coords jobs, health of all the task trackers (transfers jobs to other nodes once failures found), monitors execution percentage of jobs and resources availablility.
• Task tracker periodically heartbeat with resource information job execution stauts to jobtracker, receive and execute commands from JobTracker (start new tasks or kill existing tasks)
• Task: map task and reduce task

Question 6

Which of the following maps input k,v pairs to a set of intermediate k,v pairs?

A. Mapper
B. Reducer
C. Both A and B

Answer 6

A. Mapper

Question 7

Q2: What is the correct sequence of data flow in MapReduce?

a. InputFormat
b. Mapper
c. Combiner
d. Reducer
e. Partitioner
f. OutputFormat

A. abcdfe
B. abcedf
C. acdefb
D. abcdef

Answer 7

B. abcedf

Question 8

Q3: The total number of partitioners is equal to

A. The number of reducers
B. The number of mappers
C. The number of combiners
D. All of the above

Answer 8

A. The number of reducers