Spark Core Concepts

Question 1

What is Apache Spark at a high level?

Answer 1

A distributed data processing engine that executes computations in parallel across a cluster, supporting batch, streaming, SQL, and ML workloads.

Question 2

How does Databricks relate to Spark?

Answer 2

Databricks provides a managed, optimized Spark runtime with additional services like notebooks, jobs, Delta Lake, and governance tooling.

Question 3

What is a Spark application on Databricks?

Answer 3

A program (notebook, job, script) that uses Spark to process data on a Databricks cluster.

Question 4

What are the two main node roles in a Spark cluster?

Answer 4

The driver node, which coordinates the application, and worker nodes, which execute tasks on partitions of the data.

Question 5

What is the driver in Spark?

Answer 5

The process that runs the main application code, builds the logical plan, and schedules tasks on worker nodes.

Question 6

What is an executor in Spark?

Answer 6

A process running on a worker node that executes tasks and holds data partitions in memory or on disk.

Question 7

What is a Spark RDD (Resilient Distributed Dataset)?

Answer 7

An immutable, distributed collection of records that can be processed in parallel and recomputed from lineage information.

Question 8

Why are RDDs rarely used directly in Databricks for most workloads now?

Answer 8

Higher-level APIs like DataFrames and Spark SQL provide better optimization, safety, and convenience for structured data.

Question 9

What is a Spark DataFrame?

Answer 9

A distributed, tabular data structure with named columns and a schema, similar to a table in a relational database.

Question 10

How do DataFrames relate to Spark SQL?

Answer 10

Spark SQL queries operate on DataFrames, and DataFrames can be registered as temporary views to be queried with SQL.

Question 11

Why are DataFrames usually preferred over RDDs in Databricks?

Answer 11

They enable the Catalyst optimizer to plan and optimize queries, often yielding better performance with less code.

Question 12

What is a transformation in Spark?

Answer 12

An operation that defines a new dataset from an existing one, such as select, filter, map, or join, without triggering execution.

Question 13

What is an action in Spark?

Answer 13

An operation that triggers execution and returns a result to the driver or writes data out, such as count, collect, show, or write.

Question 14

What does it mean that Spark has lazy evaluation?

Answer 14

Transformations build a logical plan but nothing is actually executed until an action is called, allowing global optimization.

Question 15

What is a DAG (Directed Acyclic Graph) in Spark?

Answer 15

A graph of stages and operations that represent the logical execution plan of transformations leading up to an action.

Question 16

Why does Spark use DAGs instead of a fixed MapReduce pattern?

Answer 16

DAGs allow more complex multi-stage workflows and better optimization across chained transformations.

Question 17

What is a stage in Spark execution?

Answer 17

A set of tasks that can be executed without reshuffling data, separated by shuffle boundaries in the DAG.

Question 18

What is a task in Spark execution?

Answer 18

A unit of work that processes one partition of the data for a specific stage on an executor.

Question 19

What is a partition in Spark?

Answer 19

A chunk of the dataset that is processed as a unit by a single task on one executor, supporting data parallelism.

Question 20

Why is the number and size of partitions important?

Answer 20

Too few partitions underutilize the cluster; too many create overhead in scheduling and task management.

Question 21

What is a shuffle in Spark?

Answer 21

A data movement operation where records are redistributed across partitions, typically for joins, groupBy, or aggregations by key.

Question 22

Why are shuffles expensive operations?

Answer 22

They involve network transfer, disk I/O for intermediate data, and additional coordination, often dominating job runtime.

Question 23

How can you reduce shuffles in Spark jobs?

Answer 23

By avoiding unnecessary groupBy/join operations, pre-partitioning data appropriately, and reusing partitioning where possible.

Question 24

What is a wide transformation?

Answer 24

A transformation that requires data from many partitions, such as groupByKey, and usually triggers a shuffle.

Question 25

What is a narrow transformation?

Answer 25

A transformation like map or filter where each output partition depends on data from a single input partition, avoiding shuffles.

Question 26

What is caching (persisting) in Spark?

Answer 26

Storing intermediate DataFrames or RDDs in memory (and optionally disk) so they can be reused without recomputation.

Question 27

Why is caching useful in Databricks notebooks?

Answer 27

Interactive analysis often reuses the same intermediate data; caching speeds up subsequent actions on that data.

Question 28

What is the difference between `cache()` and `persist()` in Spark?

Answer 28

`cache()` uses a default storage level (usually memory-only), while `persist()` allows specifying different storage levels (e.g., memory-and-disk).

Question 29

Why should caching be used judiciously?

Answer 29

Caching too many or very large datasets can exhaust executor memory and lead to spills or eviction of useful data.

Question 30

What is a broadcast variable or broadcast join?

Answer 30

A mechanism to send a small dataset to all executors so that large datasets can be joined locally without shuffling both sides.

Question 31

When is a broadcast join appropriate in Databricks?

Answer 31

When one side of the join is small enough to fit in memory on each executor, significantly reducing shuffle cost.

Question 32

What is the Catalyst optimizer?

Answer 32

Spark SQL’s query optimizer that analyzes logical plans, applies rules, and generates optimized physical execution plans.

Question 33

Why is understanding Catalyst helpful even if you write only SQL/DataFrames?

Answer 33

It explains why certain query patterns are faster, how filters and projections are pushed down, and when joins or shuffles occur.

Question 34

What is Tungsten in Spark runtime?

Answer 34

An optimization project focusing on memory management and code generation for efficient binary processing of data.

Question 35

How does Databricks enhance Spark’s optimizer/runtime?

Answer 35

Through runtime improvements, cost-based optimizations, and Delta Lake-specific features like data skipping and file pruning.

Question 36

What is a SparkSession in Databricks?

Answer 36

The entry point for Spark functionality, exposed as 'spark' in Databricks notebooks and used to create DataFrames, run SQL, and manage configs.

Question 37

How do you typically create a DataFrame from a Delta or Parquet table in Databricks?

Answer 37

By using `spark.read.format('delta' or 'parquet').load(path)` or `spark.table('db.table_name')`.

Question 38

What is the difference between `DataFrame.show()` and `DataFrame.collect()`?

Answer 38

`show()` prints a limited number of rows for inspection, while `collect()` retrieves all results to the driver, which can be dangerous for large datasets.

Question 39

Why is calling `collect()`on large datasets a pitfall?

Answer 39

It can overwhelm driver memory and crash the application; large results should be written to storage or inspected with limited samples.

Question 40

What is the effect of using `display()` in Databricks notebooks?

Answer 40

It triggers an action similar to `show()` but with richer visualization; it still executes a job under the hood.

Question 41

How can you inspect the physical plan of a DataFrame or SQL query in Databricks?

Answer 41

By using `df.explain()` or `EXPLAIN` in SQL, optionally with 'EXTENDED' or 'CODEGEN' flags to see detailed plans.

Question 42

Why is examining `explain()` output useful for engineers?

Answer 42

It reveals where scans, filters, joins, and shuffles occur, guiding performance tuning and schema or query changes.

Question 43

What is a good mental model for Spark on Databricks from a data engineer’s perspective?

Answer 43

Write transformations in DataFrames/SQL, let Spark build a DAG, be conscious of partitions and shuffles, cache deliberately, and always think about what triggers actions and where data moves.