spark

Question 1

coomon featurs in hadoop and spark

Answer 1

statless/ memory less
no randomization

Question 2

hadoop features

Answer 2

disk based
if machine fail- uses replica

Question 3

spark features

Answer 3

uses RDD’s
stores data in memory
laxy execution
lineage traking

Question 4

transformation opperations

Answer 4

filter
mapPartition and other maps
sample
reduceby key
distinctgeoupbykey

Question 5

action operatings

Answer 5

reduce
aggregation
take sample
countbykey

Question 6

wide dependency

Answer 6

reduce by key

Question 7

diffrence between spark RDD and SQL filter function

Answer 7

Spark SQL filter function is more efficient

Spark SQL understands the logic while Spark RDD doesn’t have access to it

Question 8

Key difference between Spark RDD and Spark SQL

Answer 8

Spark SQL can achieve higher performance since it understands the logic of the opertaitons

Spark SQL employs the catalyst query optimizer

Spark SQL is awar of the data schem while spark RDD is schema agnostic

Question 9

operations

Answer 9

filter
projection
cross product
aggregation
union
intersection