Databricks Interview Prep - streaming & Pipeline Design

Question 1

Batch vs Streaming

When should you choose streaming over batch processing?

Answer 1

Use streaming when:
Low latency is required (near real-time insights)
Continuous data arrival (events, logs, IoT)
Use batch when:
Latency is not critical
Data arrives in bulk
👉 Most systems today use hybrid (Lambda/Kappa-style) architectures.

Question 2

Structured Streaming

What is Structured Streaming in Databricks?

Answer 2

A high-level API built on Spark
Treats streaming as incremental batch processing
👉 Same code works for batch and streaming → unified model.

Question 3

Micro-Batch vs Continuous Processing

What is the difference between micro-batch and continuous processing?

Answer 3

Micro-batch: processes data in small intervals (default)
Continuous: processes data row-by-row (lower latency, less used)
👉 Most production systems use micro-batch for stability.

Question 4

Checkpointing

Why is checkpointing critical in streaming pipelines?

Answer 4

Stores progress (offsets, state)
Enables recovery after failure
👉 Without checkpointing → duplicate or lost data.

Question 5

Exactly-Once Semantics

How does Databricks achieve exactly-once processing?

Answer 5

Checkpointing tracks processed data
Delta ensures transactional writes
👉 Combination ensures:
No duplicates
No data loss (in most scenarios)

Question 6

Idempotent Pipelines

How do you design an idempotent data pipeline?

Answer 6

Techniques:
Use MERGE instead of INSERT
Deduplicate using keys
Track processed files/events
👉 Essential for retries and failure recovery.

Question 7

Auto Loader

What is Auto Loader and when should you use it?

Answer 7

Incremental file ingestion tool (cloudFiles)
Efficiently detects new files
Use when:
Large-scale file ingestion (e.g., millions of files)
👉 Preferred over manual ingestion.

Question 8

Auto Loader vs COPY INTO

What is the difference between Auto Loader and COPY INTO?

Answer 8

Auto Loader: streaming + scalable + incremental
COPY INTO: batch-oriented
👉 Auto Loader is better for:
Continuous ingestion
Large volumes

Question 9

Schema Evolution in Streaming

How do you handle schema changes in streaming pipelines?

Answer 9

Enable schema evolution carefully
Validate schema changes
Use a schema registry if needed
👉 Uncontrolled evolution can break downstream pipelines.

Question 10

Late-Arriving Data

How do you handle late-arriving data in streaming?

Answer 10

Use watermarking
Allow updates via MERGE
👉 Ensures correctness without infinite waiting.

Question 11

Watermarking

What is watermarking in streaming?

Answer 11

Defines how long to wait for late data
Helps manage state and memory
👉 Tradeoff:
Too short → data loss
Too long → high memory usage

Question 12

Trigger Modes

What are trigger modes in Structured Streaming?

Answer 12

Processing time (e.g., every 1 min)
Once (batch-like execution)
Available now
👉 Controls frequency of execution.

Question 13

Stateful vs Stateless Processing

What is the difference between stateful and stateless streaming?

Answer 13

Stateless → each record independent
Stateful → depends on past data (e.g., aggregations)
👉 Stateful requires more memory and checkpointing.

Question 14

Handling Duplicates

How do you handle duplicate data in pipelines?

Answer 14

Use unique keys
Deduplicate using window functions
Use MERGE logic
👉 Duplicate handling is critical in distributed systems.

Question 15

CDC (Change Data Capture)

How do you implement CDC pipelines in Databricks?

Answer 15

Use MERGE INTO
Track inserts/updates/deletes
Use Delta Change Data Feed if available

👉 Enables incremental updates instead of full reloads.

Question 16

Designing a File Ingestion Pipeline

How would you design a pipeline to ingest millions of files daily?

Answer 16

Use Auto Loader
Store raw data in Bronze
Enable schema evolution
Optimize with checkpointing

👉 Avoid listing all files manually → scalability issue.

Question 17

Failure Handling

How do you handle failures in data pipelines?

Answer 17

Retry mechanisms
Idempotent design
Checkpointing
Logging and alerting
👉 Pipelines must be resilient by design.

Question 18

Incremental vs Full Load

Why is incremental processing preferred over full load?

Answer 18

Faster
Cheaper
Scales better
👉 Full load is only for:
Initial load
Rare reprocessing scenarios

Question 19

Orchestration (Databricks Jobs)

How do you orchestrate pipelines in Databricks?

Answer 19

Use Jobs / Workflows
Define task dependencies
Schedule runs
👉 Supports:
Retry logic
Monitoring

Question 20

Bronze Layer Design

What is the best practice for Bronze layer ingestion?

Answer 20

Append-only
Minimal transformation
Store raw data
👉 Ensures replayability.

Question 21

Silver Layer Design

What transformations belong in the Silver layer?

Answer 21

Data cleaning
Deduplication
Joins
👉 Creates a trusted dataset.

Question 22

Gold Layer Design

What is the purpose of Gold layer tables?

Answer 22

Business-level aggregations
Optimized for BI queries
👉 Should be:
Clean
Fast
Easy to query

Question 23

Data Quality Checks

Where should data quality checks be applied?

Answer 23

Primarily in Silver layer
Optionally in Bronze for critical validations
👉 Prevents bad data propagation.

Question 24

Streaming vs Batch Cost Tradeoff

What is the cost tradeoff between streaming and batch?

Answer 24

Streaming → always running → higher cost
Batch → scheduled → cheaper
👉 Choose based on latency requirements.

Question 25

# End-to-End Pipeline Design How would you design a robust end-to-end pipeline in Databricks?

Answer 25

Ingest with Auto Loader → Bronze Clean + deduplicate → Silver Aggregate → Gold Use MERGE for idempotency Add monitoring + retries 👉 Key principles: Scalability Reliability Maintainability