Spark SQL & Catalyst Optimizer

Question 1

What is Delta Lake at a high level?

Answer 1

An open table storage format that adds ACID transactions, schema enforcement, and other data management features on top of files in cloud object storage.

Question 2

How does Delta Lake relate to Parquet files?

Answer 2

Delta Lake stores data in Parquet files but adds a transaction log and metadata layer that tracks versions and operations on those files.

Question 3

Where are Delta Lake tables typically stored on Databricks?

Answer 3

On cloud object storage (such as S3, ADLS, or GCS), accessed via DBFS or a catalog, with a _delta_log directory that holds transaction logs.

Question 4

What is the _delta_log directory in a Delta table?

Answer 4

A folder that contains JSON and checkpoint files describing all committed transactions, schema changes, and file-level metadata for the table.

Question 5

What does it mean that Delta Lake provides ACID transactions?

Answer 5

Operations on Delta tables are atomic, consistent, isolated, and durable, ensuring readers see consistent snapshots and writes are either all applied or not at all.

Question 6

What is optimistic concurrency control in Delta Lake?

Answer 6

A mechanism where writers assume they will not conflict, but check the transaction log at commit time and retry or fail if conflicting changes have occurred.

Question 7

Why is optimistic concurrency control well-suited to Delta on object storage?

Answer 7

It avoids centralized locks and coordinates through the transaction log, working with the eventual consistency and immutability of object storage.

Question 8

What is snapshot isolation in Delta Lake?

Answer 8

A property that each query sees a consistent snapshot of the table as of a particular version, unaffected by concurrent writes that commit later.

Question 9

How does Delta Lake achieve snapshot isolation for readers?

Answer 9

Readers use the transaction log to build a view of which data files are valid for a particular table version and read only those files.

Question 10

What is schema enforcement in Delta Lake?

Answer 10

The requirement that data written to a Delta table matches its defined schema, preventing incompatible data from being appended by default.

Question 11

Why is schema enforcement important in a data lake setting?

Answer 11

It prevents silent corruption or drifting of schemas that would otherwise be easy when writing raw Parquet files directly to object storage.

Question 12

What is schema evolution in Delta Lake?

Answer 12

The ability to change a table’s schema over time (e.g., adding columns) in a controlled way, tracked via the transaction log.

Question 13

What is a common safe form of schema evolution in Delta tables?

Answer 13

Adding new nullable columns or fields while leaving existing columns compatible with prior versions.

Question 14

Why must schema evolution be intentional and not automatic?

Answer 14

Uncontrolled changes can break downstream readers, so explicit evolution helps maintain compatibility and governance.

Question 15

What is the difference between a Delta table and a raw Parquet folder?

Answer 15

A Delta table has a transaction log that tracks adds/removes and schema changes, enabling ACID operations and time travel; a Parquet folder is just files without table semantics.

Question 16

What does ‘table version’ mean in Delta Lake?

Answer 16

An integer that identifies a specific committed state of the table, incremented with each transaction.

Question 17

How does time travel work in Delta tables?

Answer 17

By reading the table as of a previous version or timestamp, using the transaction log to reconstruct that historical snapshot.

Question 18

What are typical use cases for Delta time travel?

Answer 18

Auditing historical data, debugging changes, recovering from accidental modifications, and reproducing past experiments.

Question 19

What is a Delta table ‘path-based’ table?

Answer 19

A table defined directly on a storage path, accessed via its filesystem location rather than via a catalog name.

Question 20

What is a ‘managed’ or ‘catalog-registered’ Delta table?

Answer 20

A Delta table registered in a metastore or Unity Catalog with a logical name, where the catalog tracks its location, schema, and permissions.

Question 21

Why are catalog-registered Delta tables often preferable in production?

Answer 21

They provide consistent naming, governance, permissions, and lineage, decoupling logical access from raw storage paths.

Question 22

What is a Delta Lake ‘transaction’?

Answer 22

A set of operations (such as adding or removing files, schema changes) that are committed as a single atomic change to the table’s log.

Question 23

How is a Delta transaction committed in the log?

Answer 23

By writing a new JSON entry in the _delta_log and possibly updating checkpoints, describing which files are added or removed and any metadata changes.

Question 24

What is a checkpoint in the Delta transaction log?

Answer 24

A Parquet file that periodically summarizes the log up to a version, allowing faster reconstruction of table state than replaying all JSON logs.

Question 25

Why are checkpoints important for large, long-lived Delta tables?

Answer 25

They speed up startup and reading performance by avoiding scanning many individual log files.

Question 26

What is data skipping in Delta Lake?

Answer 26

An optimization where statistics about columns in each file (such as min/max) are stored and used to avoid reading files that cannot satisfy query predicates.

Question 27

How does data skipping improve performance?

Answer 27

By allowing the engine to skip entire files whose statistics show they do not contain values needed for a query, reducing I/O and CPU work.

Question 28

What is Z-ordering (ZORDER) in Delta Lake?

Answer 28

A technique to co-locate related values in the same set of files, improving data skipping effectiveness for specific columns or queries.

Question 29

When might you use ZORDER on a Delta table?

Answer 29

When you frequently filter or join on certain columns and want to improve clustering and skipping for those columns.

Question 30

What is VACUUM in Delta Lake?

Answer 30

A maintenance command that removes old data files that are no longer referenced by the transaction log after a retention period.

Question 31

Why is VACUUM important for Delta tables?

Answer 31

To prevent build-up of unused files and control storage costs, while respecting retention settings for time travel and rollback.

Question 32

What is the retention period in Delta VACUUM?

Answer 32

The duration for which old file versions are kept to support time travel and rollback; files older than this can be removed by VACUUM.

Question 33

Why must the retention period not be set too low?

Answer 33

Too low retention may eliminate the ability to time-travel or recover from recent mistakes, reducing safety and auditability.

Question 34

What is the default file format used by Delta tables?

Answer 34

Parquet for data files, with JSON and Parquet for the transaction log.

Question 35

How do Delta tables support both batch and streaming workloads?

Answer 35

The transactional log makes it possible to read and write Delta tables in batch jobs and as streaming sources/sinks with consistent semantics.

Question 36

Why is it beneficial to use the same Delta tables for batch and streaming?

Answer 36

It unifies historical and real-time data, reducing duplication and ensuring consistent semantics across processing modes.

Question 37

What happens if multiple writers update a Delta table at the same time?

Answer 37

Each writer attempts an optimistic commit; if a conflicting commit is detected, one writer must retry or fail to preserve ACID guarantees.

Question 38

Why is it important for a data engineer to understand Delta Lake internals conceptually?

Answer 38

Because it informs how to design robust pipelines, reason about performance and concurrency, and debug issues involving versions or schema changes.

Question 39

In one sentence, what is the core mental model for Delta Lake on Databricks?

Answer 39

Delta Lake turns folders of Parquet files on object storage into ACID tables with transaction logs, schema control, and time travel, forming the foundation of the lakehouse.