Spark commands

Question 1

Name 6 action commands we learned in class

Answer 1

.colllect()
.take(num=filestaken)
.foreach()
.count()
.countByValue()
.saveAsTextFile()

Question 2

Name 9 transformation commands we learned in class

Answer 2

• filter
• map
• flatMap
• sample
• distinct
• union
• intersection
• subtract
• cartesian

Question 3

What is the pyspark syntax to start a spark context

Answer 3

conf = SparkConf().setAppName('example').setMaster('local[*]')
sc = SparkContext(conf=conf)

Question 4

What are different syntax to load a file into a Spark environment?

Answer 4

df = spark.read.load(‘FOO’, format=’FILEFORMAT’, inferSchema = ‘true’, header = ‘true’)

alternatively you can load a file via:

df = spark.read.json('FOO')
df = spark.read.csv('FOO')

Question 5

What are the five categories of spark?

Answer 5

1) SparkCore – SparkContext
2) SparkSQL – SparkSession
3) Spark Streaming
4) MLlib
5) graphx