Cluster & Compute Management

Question 1

What is a Databricks cluster at a high level?

Answer 1

A set of compute resources (driver and workers) configured with a runtime and libraries, used to run Spark jobs, notebooks, and SQL workloads.

Question 2

What are the two main types of clusters in Databricks?

Answer 2

All-purpose clusters for interactive use and job clusters for scheduled or one-off job runs.

Question 3

When are all-purpose clusters typically used?

Answer 3

For interactive development, exploratory analysis, and collaborative notebook work.

Question 4

When are job clusters typically used?

Answer 4

For production jobs and workflows, where each run gets a clean, ephemeral cluster with defined configuration.

Question 5

Why are job clusters recommended for production workloads?

Answer 5

They avoid interference between users, ensure reproducible environments, and can be sized exactly for each job’s needs.

Question 6

What is the driver node’s role on a Databricks cluster?

Answer 6

It runs the Spark driver process, coordinates execution, and usually holds the notebook state or main application logic.

Question 7

What is a worker node’s role on a Databricks cluster?

Answer 7

It runs executor processes that perform actual data processing tasks on partitions and store intermediate data.

Question 8

What does autoscaling mean for a Databricks cluster?

Answer 8

The ability to automatically increase or decrease the number of worker nodes based on workload demand within configured bounds.

Question 9

Why is autoscaling useful?

Answer 9

It can improve resource utilization and cost efficiency by scaling up under load and scaling down when idle or lightly loaded.

Question 10

What is a reasonable upper limit on autoscaling tied to?

Answer 10

Expected peak concurrency and data volume; too high a limit can cause excessive costs or strain downstream systems.

Question 11

What is a cluster policy in Databricks?

Answer 11

An admin-defined set of constraints and defaults on cluster configurations to enforce best practices and cost controls.

Question 12

Why are cluster policies important in larger organizations?

Answer 12

They standardize runtimes, node types, autoscaling ranges, and security settings, reducing misconfiguration and cost surprises.

Question 13

What are some common parameters controlled by cluster policies?

Answer 13

Instance families and sizes, autoscaling bounds, runtime versions, spot vs on-demand usage, and permission scopes.

Question 14

What is Databricks Runtime (DBR)?

Answer 14

A curated and optimized Spark runtime with bundled libraries and performance improvements provided by Databricks.

Question 15

Why is it important to choose a specific DBR version rather than ‘latest’?

Answer 15

Pinning versions ensures reproducibility and avoids unexpected behavior changes when runtimes are updated.

Question 16

What is the difference between CPU and GPU clusters in Databricks?

Answer 16

CPU clusters use standard compute nodes; GPU clusters use nodes with GPUs for accelerated ML, DL, or compute-intensive tasks.

Question 17

When might you choose a GPU cluster?

Answer 17

For training deep learning models or other GPU-accelerated libraries where workloads benefit from large parallelism.

Question 18

What is spot (or preemptible) capacity in cluster configuration?

Answer 18

Discounted instances that can be reclaimed by the cloud provider, offering lower cost at the expense of potential interruptions.

Question 19

Why is mixing spot and on-demand nodes sometimes advantageous?

Answer 19

It can lower cost while retaining some resiliency, as critical tasks can run on on-demand nodes while extra capacity uses cheaper spot nodes.

Question 20

What is cluster auto-termination?

Answer 20

A setting that automatically terminates a cluster after a period of inactivity, preventing idle compute charges.

Question 21

Why is auto-termination essential for all-purpose clusters?

Answer 21

Interactive clusters are easy to forget; auto-termination avoids paying for unused resources when work stops.

Question 22

What is the relationship between cluster size and parallelism?

Answer 22

Larger clusters (more cores) can run more tasks in parallel, but require enough data and proper partitioning to be fully utilized.

Question 23

Why can a cluster that is too large be inefficient?

Answer 23

If workloads and data sizes are too small, many cores sit idle, wasting cost without speeding up jobs significantly.

Question 24

What is executor memory used for?

Answer 24

Holding data partitions, cached DataFrames, and intermediate results during computations.

Question 25

What happens when executors run out of memory?

Answer 25

Spark spills data to disk, which slows down jobs, or may trigger out-of-memory errors and task failures if severe.

Question 26

What is the effect of very small partitions on performance?

Answer 26

They increase scheduling and overhead, leading to less efficient use of the cluster and longer runtimes.

Question 27

What is the effect of very large partitions on performance?

Answer 27

They can cause skew, long-running tasks, and higher memory pressure, risking spills or failures on specific executors.

Question 28

What configuration controls partition sizes in Spark?

Answer 28

Settings like spark.sql.shuffle.partitions and input partitioning of source data influence the number and size of partitions.

Question 29

Why should spark.sql.shuffle.partitions be tuned?

Answer 29

The default may be too high or low for your workload; tuning can reduce unnecessary shuffles and improve performance.

Question 30

What is data skew and how does it show up on a cluster?

Answer 30

Uneven distribution of data across partitions; it appears as a few tasks taking much longer than others and underutilized workers.

Question 31

What strategies help mitigate skew on Databricks?

Answer 31

Salting keys, pre-aggregation, using broadcast joins where appropriate, and redesigning partitioning or join keys.

Question 32

Why is reading from Delta tables often faster than raw Parquet on Databricks?

Answer 32

Delta’s transaction log, statistics, and data skipping enable more efficient file pruning and planning compared to plain folders.

Question 33

What is Spark caching and how does it relate to cluster memory?

Answer 33

Caching stores DataFrames in executor memory for reuse; if memory is limited, caching may cause spills or eviction of useful data.

Question 34

When is it appropriate to cache a DataFrame on Databricks?

Answer 34

When it is reused multiple times across actions and fits comfortably within available memory without crowding out other workloads.

Question 35

Why should unnecessary caching or persist calls be removed?

Answer 35

They consume memory and can degrade performance if cached data is rarely reused or too large.

Question 36

What is the purpose of 'explain' plans for performance tuning?

Answer 36

They show the physical operations (scans, filters, joins, shuffles) and help identify bottlenecks or unnecessary work.

Question 37

What metrics or tools can be used to analyze job performance on Databricks?

Answer 37

Spark UI, Databricks job run details, cluster metrics dashboards, and logs for tasks, executors, and stages.

Question 38

Why is the Spark UI important for tuning?

Answer 38

It reveals stage runtimes, shuffle volumes, skewed tasks, and executor utilization, guiding where to optimize code or configuration.

Question 39

How can you reduce data scanned per query in Databricks?

Answer 39

By using partition pruning, data skipping, ZORDER on Delta tables, and selecting only necessary columns.

Question 40

Why should SELECT * be avoided in production ETL and analytics?

Answer 40

It reads more columns than needed, increases I/O and network usage, and makes schema evolution riskier.

Question 41

What is a good practice for isolating heavy workloads on Databricks?

Answer 41

Run them on dedicated job clusters or separate SQL warehouses so they do not impact interactive or latency-sensitive workloads.

Question 42

How does cluster concurrency affect performance?

Answer 42

Too many concurrent jobs can cause resource contention, leading to longer runtimes and higher failure rates; concurrency limits help maintain stability.

Question 43

What is a reasonable approach to cluster sizing on Databricks?

Answer 43

Start with a modest configuration, measure utilization and runtimes, and adjust size and autoscaling based on empirical evidence rather than guesswork.

Question 44

In one sentence, what is the core mental model for cluster and performance tuning on Databricks?

Answer 44

Treat clusters as flexible, experiment-driven resources: tune size, autoscaling, partitions, and storage layout so Spark jobs fully use the cluster without wasting compute or suffering from skew and spills.