Apache Spark Core and Structured Streaming by Amit Ranjan

Question 1

What is SLA in context of ETL?
Give example.

Answer 1

SLA = Service Level Agreement
Example of SLA:
“Your data must be available in transformed way till 6 in the morning.”

Question 2

Data Processing framework should be able to keep the latency under control, even when data volume scales.
i.e., Latency should not scale linearly with data _ .

Answer 2

volume

This can be achieved by applying vertical and horizontal scalability

Question 3

Data Processing Framework should be resilient towards transient network failures.
To this end, the restarting of a failed job should be _ .

Answer 3

automatic

Question 4

Data Processing Framework should support integration (ingestion) with diverse set of data sources.
Name some.

Answer 4

• file stored in HDFS/S3 storage
• JSON file from REST api
• data from a SQL/NoSQL database

Similarly the DP Framework should support writing to diverse set of data targets.

Question 5

Data Processing Framework should support all kinds of data _ :
* batch
* streaming
* near real-time

Answer 5

ingestion

Additionally, DP Framework should be easy to upgrade with backward compatibility, easy to customize and extend and operationally smooth enough to hide implementational complexities.

Question 6

Spark is in-memory distributed data processing framework, which is an improvement over _ framework.

Answer 6

map-reduce

Question 7

RDD full form?

Answer 7

Resilient Distibuted Dataset

RDD is an immutable data structure. It doesn’t contain data, only the lineage information.

Question 8

Partitions of RDD are distributed across the _ of the cluster.

Answer 8

nodes

Question 9

When an RDD is created or transformed, the associated _ (dependency) information is updated and kept in-memory by Spark.

Answer 9

lineage

Lineage information is essential to avoid reprocessing source RDD if one of the partitions of RDD gets lost due to failure.

Question 10

The operations on RDD can be classified into two categories.
Name them.

Answer 10

1. Transformations
2. Actions

Question 11

How do operations on RDD relate with lineage?
1. Transformation _ the lineage.
2. Action _ the result/outcome of all updates in the lineage

Answer 11

1. updates
2. demands

Question 12

Partitions of RDD are distributed across the nodes of the cluster.
Each partition of RDD is processed _ by one task of the executor core.

Answer 12

parallely (in parallel)

Question 13

Before and after Shuffling, the number of partitions of the RDD _ .

(stay same/can change)

Answer 13

can change

Question 14

Lineage information is essential to avoid reprocessing entire source RDD if one of the partitions of RDD gets lost due to failure.

When one partition of RDD fails, Spark launches a new task which uses lineage information of the RDD to traverse back to read only the _ (avoiding the need to reprocess entire RDD) and rebuild the partition using transformations stored in lineage.

Answer 14

source partition

Question 15

Adding an extra transformation to an RDD does not result in Spark computation, unless it causes a _ .

Answer 15

shuffle

Question 16

Transformations that cause shuffling are known as _ .

Answer 16

“wide transformations”

Thus, narrow transformations do not lead to any extra performance penalty.

Question 17

Spark Driver program is responsible for instantiating a _ , which provides a single unified entry point to all of Spark’s functionality (API).

An entry point is the main object through which you access Spark’s capabilities like
1. Core RDD APIs
2. DataFrame/Dataset APIs
3. SQL queries
4. Streaming
5. Hive support

Answer 17

SparkSession

It represents a single Spark Application, i.e., one spark application can have only one SparkSession.

Question 18

SparkSession is responsible for sending extra JAR files for _ on executors (worker nodes).

Answer 18

configuration

Question 19

Write Scala code to create a SparkSession.
Specify app name: “SparkApp”.
Specify master node as “local”.

Answer 19

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("SparkApp")
.master("local[*]")
.getOrCreate()

Question 20

Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings with 10 partitions.

path = “hdfs:/path/to/file”

Answer 20

// Access SparkContext from SparkSession
val sc = spark.sparkContext 

// File path can be HDFS, local FS, or any Hadoop-supported URI 
val filePath = args(0) 

// e.g. "hdfs://namenode:8020/path/to/file.txt" // or "file:///path/to/localfile.txt" 

// Read file into RDD[String] with 10 partitions 
val textRDD = sc.textFile(filePath, minPartitions = 10) 

// Example action: count lines 
val lineCount = textRDD.count() println(s"Total lines in file: $lineCount") 

// Stop SparkSession 
spark.stop()

Question 21

Add a JAR dependency for all tasks to be executed on this SparkContext in the future.

Answer 21

// Access SparkContext from SparkSession
val sc = spark.sparkContext

// JAR File path can be HDFS, local FS, or any Hadoop-supported URI 
val filePath = args(0) 

sc.addJar(filePath)

Note: If a jar is added during execution, it will not be available until the next TaskSet starts.

Question 22

_ provides resources to your Spark Application.

Answer 22

Cluster Manager

resources like network (bandwidth), cpu, ram

Question 23

Cluster Manager is not a part of the Spark Application.
It’s an external service used by Spark Application to acquire resources on the cluster.
Name few examples of Cluster Managers.

Answer 23

• Standalone Manager
• Mesos
• Yarn (in Hadoop)
• Kubernetes

Question 24

Driver node connects with Cluster Manager to request _ for executors, until the SparkSession acquires them.

Answer 24

resources

Question 24

The node where SparkSession gets created is known as the _ .

Answer 24

Driver Node

Question 25

Number of Tasks for processing a RDD = Number of _ it has.

Answer 25

partitions

Question 26

The application code defined by JAR files is sent over network to each _ (JVM process).

Answer 26

executor ## Footnote This is handled by Spark internally; no code required.