Databricks Interview Prep - Scenarios-based questions

Question 1

End-to-End Pipeline Design

Answer 1

Design a pipeline to ingest and process millions of files daily.

• Use Auto Loader for scalable ingestion
• Store raw data in Bronze (append-only)
• Clean + deduplicate in Silver (MERGE)
• Aggregate in Gold

Add:

• Checkpointing for reliability
• OPTIMIZE for performance

👉 Key focus: scalability + idempotency + maintainability

Question 2

Debugging Slow Pipeline

Answer 2

A pipeline that used to run in 10 minutes now takes 1 hour. What do you check?

• Data growth (volume increased?)
• Small file problem
• Data skew
• Execution plan (shuffles?)

Fix:

• OPTIMIZE + ZORDER
• Repartition
• Broadcast joins

👉 Always identify the biggest bottleneck first.

Question 3

Handling Duplicate Data

Answer 3

Your pipeline is producing duplicate records. How do you fix it?

• Identify unique keys
• Use MERGE instead of INSERT
• Deduplicate using window functions

👉 Ensure pipeline is idempotent.

Question 4

Late-Arriving Data Scenario

Answer 4

Data arrives late in a streaming pipeline. How do you handle it?

• Use watermarking
• Allow updates via MERGE
• Adjust latency vs accuracy tradeoff

👉 Balance:

• Correctness vs performance

Question 5

Data Skew Issue

Answer 5

One Spark task is much slower than others. What’s happening?

Likely data skew:

• One partition has much more data

Fix:

• Salting keys
• Repartitioning
• Skew join optimization

👉 Common real-world bottleneck.

Question 6

Small File Problem in Production

Answer 6

Your Delta table has thousands of small files. What do you do?

• Run OPTIMIZE (compaction)
• Adjust ingestion batch size
• Use proper partitioning

👉 Prevent future issues, not just fix current ones.

Question 7

Choosing Partition Strategy

Answer 7

How do you choose a partition column?

• Based on query patterns
• Low cardinality (e.g., date)

Avoid:

• High-cardinality columns

👉 Partitioning should match access patterns, not guesswork.

Question 8

Streaming vs Batch Decision

Answer 8

How do you decide between streaming and batch?

• Streaming → low latency needed
• Batch → cost efficiency

👉 Most systems combine both.

Question 9

Handling Pipeline Failure

Answer 9

A pipeline fails midway. How do you ensure no data loss or duplication?

• Use checkpointing
• Design idempotent logic (MERGE)
• Retry safely

👉 Recovery must be automatic and safe.

Question 10

CDC Pipeline Design

Answer 10

How would you design a CDC pipeline?

• Capture changes (insert/update/delete)
• Use MERGE INTO
• Optionally use Delta Change Data Feed

👉 Avoid full reloads → efficiency.

Question 11

Optimizing Join Performance

Answer 11

A join operation is very slow. How do you optimize it?

• Use broadcast join if possible
• Reduce data before join (filter early)
• Check partitioning

👉 Joins are often the biggest bottleneck.

Question 12

Reprocessing Historical Data

Answer 12

You need to reprocess 6 months of data. How do you approach it?

• Use Bronze as source of truth
• Rebuild Silver/Gold

👉 Avoid re-ingestion → faster and safer.

Question 13

Handling Schema Changes

Answer 13

A new column appears in incoming data. What do you do?

• Enable schema evolution (if safe)
• Validate downstream impact

👉 Schema changes should be controlled, not automatic everywhere.

Question 14

Designing for Scalability

Answer 14

How do you design a pipeline that scales over time?

• Use distributed processing (Spark)
• Avoid small files
• Partition data properly

👉 Always assume data will grow.

Question 15

Ensuring Data Quality

Answer 15

How do you ensure data quality in pipelines?

• Validate schema
• Deduplicate
• Apply rules in Silver layer

👉 Catch issues early before reaching Gold.

Question 16

Cost Optimization

Answer 16

How do you reduce cost in Databricks pipelines?

• Use batch instead of streaming when possible
• Optimize file sizes
• Turn off idle clusters

👉 Balance performance vs cost.

Question 17

Choosing Between Databricks and Snowflake

Answer 17

When would you choose Databricks over Snowflake?

• Complex transformations
• Streaming pipelines
• ML workloads

👉 Databricks = engineering flexibility
👉 Snowflake = SQL simplicity.

Question 18

Real-Time Analytics Design

Answer 18

How would you design a real-time analytics system?

• Streaming ingestion (Auto Loader / Kafka)
• Process with Structured Streaming
• Store in Delta tables

👉 Ensure:

• Low latency
• Fault tolerance

Question 19

Handling Backfill + Streaming Together

Answer 19

How do you handle historical backfill while streaming is running?

• Run batch backfill separately
• Merge results into same table

👉 Avoid disrupting streaming pipeline.

Question 20

Monitoring Pipelines

Answer 20

How do you monitor pipeline health?

• Logs + metrics
• Job monitoring
• Alerts on failures

👉 Observability is critical in production.

Question 21

Multi-Tenant Data Design

Answer 21

How do you design a system for multiple teams using the same platform?

• Use Unity Catalog
• Separate catalogs/schemas
• Apply RBAC

👉 Ensures isolation + governance.

Question 22

Handling Large Tables

Answer 22

A table has billions of rows and queries are slow. What do you do?

• Partition properly
• Use ZORDER
• Optimize file size

👉 Reduce scan size as much as possible.

Question 23

Designing Idempotent Streaming

Answer 23

How do you ensure idempotency in streaming pipelines?

• Use checkpointing
• Deduplicate records
• Use MERGE

👉 Required for reliable streaming.

Question 24

Tradeoff: Performance vs Cost

Answer 24

How do you balance performance and cost?

• Optimize only bottlenecks
• Avoid over-engineering
• Use right cluster size

👉 Don’t optimize everything—optimize what matters.

Question 25

Thinking Like a Senior Engineer

Answer 25

What distinguishes a senior data engineer in Databricks? * Focus on system design, not just code * Understand tradeoffs * Build reliable and scalable pipelines 👉 It’s about **decision-making, not just implementation**.