Databricks Runtime & Performance Tuning

Question 1

What is Databricks Runtime (DBR)?

Answer 1

A curated, optimized Spark distribution provided by Databricks that bundles specific Spark versions, libraries, and performance improvements.

Question 2

Why should you pin a specific DBR version instead of always using the latest?

Answer 2

Pinning ensures reproducibility and stability; automatically upgrading to the latest can introduce behavior changes or incompatibilities unexpectedly.

Question 3

What is Photon in Databricks?

Answer 3

A next-generation query engine built in C++ that accelerates SQL and DataFrame workloads, particularly for Delta and Parquet data.

Question 4

When can enabling Photon be beneficial?

Answer 4

For SQL-heavy, scan-heavy, and aggregation-heavy workloads where you want lower latency and better throughput on supported operations.

Question 5

Why is it important to match DBR type (e.g., ‘Standard’ vs ‘Photon’) to workload characteristics?

Answer 5

Some workloads benefit more from Photon and vectorized execution, while others may rely on libraries or features not fully optimized by Photon.

Question 6

What is adaptive query execution (AQE) in Spark/Databricks?

Answer 6

A runtime optimization that adjusts query execution plans on the fly based on observed statistics, such as changing join strategies or shuffle partitions.

Question 7

How can AQE improve performance?

Answer 7

By coalescing small shuffle partitions, dynamically switching join types (e.g., to broadcast), and handling skew more intelligently at runtime.

Question 8

Why is input file layout important for performance on Databricks?

Answer 8

Poorly organized data can lead to many small files, skewed partitions, and inefficient scans, while well-laid-out Delta/Parquet data enables skipping and parallelism.

Question 9

What are some Delta-specific optimizations that affect performance?

Answer 9

OPTIMIZE to compact small files, ZORDER BY for better clustering, and proper partitioning to enable partition pruning.

Question 10

What is data skipping in Delta and why does it matter for performance?

Answer 10

Data skipping uses file-level statistics (e.g., min/max per column) to avoid scanning files that cannot satisfy filter predicates, reducing I/O.

Question 11

How does choosing partition columns affect performance in Databricks?

Answer 11

Good partition keys align with common filters and produce balanced partition sizes; poor keys can create hotspots or many small files.

Question 12

What is spark.sql.shuffle.partitions and why is it important?

Answer 12

A configuration controlling the default number of shuffle partitions; tuning it helps avoid too many tiny partitions or too few large ones.

Question 13

When might you reduce spark.sql.shuffle.partitions from its default?

Answer 13

When your dataset is modest or your cluster is small, to avoid overhead from scheduling an excessive number of tiny tasks.

Question 14

When might you increase spark.sql.shuffle.partitions?

Answer 14

When processing very large datasets on large clusters to provide enough parallelism and avoid huge single partitions.

Question 15

How do broadcast joins help performance in Databricks?

Answer 15

They replicate small tables to all executors so that large tables can be joined locally, avoiding expensive shuffles of both sides.

Question 16

What configuration can influence when Spark chooses broadcast joins?

Answer 16

spark.sql.autoBroadcastJoinThreshold, which sets the maximum size of a table that Spark will consider broadcasting automatically.

Question 17

Why is SELECT * a performance anti-pattern in production?

Answer 17

It reads all columns even when only a subset is needed, increasing I/O, CPU, and network usage and making schema evolution riskier.

Question 18

What is column pruning and how does it work in Spark SQL?

Answer 18

An optimization where the engine reads only the columns required by the query, skipping unnecessary data in columnar formats.

Question 19

Why is predicate pushdown important for performance?

Answer 19

It pushes filter operations down to the data scan layer so fewer rows are read from storage, reducing I/O and compute costs.

Question 20

How does caching interact with Databricks Runtime performance?

Answer 20

Caching frequently reused DataFrames can greatly speed up repeated queries, but over-caching can exhaust memory and cause spills.

Question 21

What is a good rule of thumb for caching in Databricks?

Answer 21

Cache only intermediate results that are reused multiple times, and unpersist them when they are no longer needed.

Question 22

What are common signs that a job is I/O-bound on Databricks?

Answer 22

High scan times, low CPU utilization, lots of time spent in ‘scan’ stages, and performance improving mainly with better layout or pruning.

Question 23

What are common signs that a job is CPU-bound?

Answer 23

High CPU utilization, long-running compute-heavy stages (joins, aggregations), and limited improvement from changing storage layout.

Question 24

How can you identify skew issues using the Spark UI?

Answer 24

By looking for stages where a few tasks take much longer and process much more data than others, indicating uneven partition sizes.

Question 25

What runtime strategies help mitigate data skew?

Answer 25

Key salting, pre-aggregation, using broadcast joins for small tables, and redesigning join keys or partitioning schemes.

Question 26

Why is it useful to examine explain() plans regularly?

Answer 26

They reveal where scans, shuffles, and joins occur, helping spot unnecessary complexity, missed filters, or suboptimal join orders.

Question 27

How can cluster autoscaling settings impact runtime performance?

Answer 27

Too aggressive scaling down can cause fragmentation and warm-up overhead; too conservative scaling up can leave the cluster underpowered under load.

Question 28

Why are long-running all-purpose clusters more prone to performance drift?

Answer 28

They accumulate library changes, cached state, and possible misconfigurations; ephemeral job clusters often provide more predictable performance.

Question 29

What is the relationship between DBR version and library compatibility?

Answer 29

Each DBR version supports specific Spark/Python/Scala versions and library versions; using incompatible libraries can cause runtime errors or degraded performance.

Question 30

What is a good process for runtime and performance tuning on Databricks?

Answer 30

Start with a pinned DBR and modest cluster, measure job behavior via Spark UI and metrics, then iteratively tune partitions, layout (Delta optimizations), joins, and cluster settings based on evidence.

Question 31

In one sentence, what is the core mental model for Databricks Runtime & Performance Tuning?

Answer 31

Use the optimized runtime and Delta features, then refine partitions, joins, and cluster settings based on actual job profiles instead of guessing, so you minimize shuffles and I/O while avoiding skew and memory issues.