Mental Models

Question 1

What is Databricks at a high level?

Answer 1

A unified analytics and data engineering platform built around Apache Spark that provides managed compute, collaborative notebooks, and a lakehouse-style data architecture.

Question 2

How does Databricks relate to Apache Spark?

Answer 2

Databricks is built by the creators of Spark and provides a managed, optimized Spark runtime along with additional services and tooling.

Question 3

Where does Databricks typically sit in a modern data platform?

Answer 3

As the primary compute and transformation layer on top of cloud object storage, supporting ETL, batch, streaming, analytics, and ML.

Question 4

What is the ‘lakehouse’ concept associated with Databricks?

Answer 4

A data architecture that combines the flexibility of data lakes with the reliability and performance of data warehouses using a single storage layer and table format like Delta Lake.

Question 5

What are the three core concerns Databricks tries to unify?

Answer 5

Data engineering, data science/ML, and analytics/BI on a shared platform and storage layer.

Question 6

Why is Databricks often used alongside cloud data warehouses like Snowflake or Redshift?

Answer 6

Databricks excels at heavy ETL, Spark-based processing, and ML, while warehouses often remain the primary SQL-serving and BI semantic layer.

Question 7

What is the primary storage layer underlying Databricks workloads?

Answer 7

Cloud object storage (e.g., S3, ADLS, GCS) accessed via the Databricks File System (DBFS) and table formats like Delta Lake.

Question 8

What is DBFS conceptually?

Answer 8

An abstraction that presents object storage (and some local storage) in a file system-like interface for Databricks clusters and notebooks.

Question 9

Why is it important to understand that DBFS ultimately sits on object storage?

Answer 9

Because it affects performance characteristics, file immutability, partitioning strategies, and cost patterns familiar from data lakes.

Question 10

What is a Databricks workspace?

Answer 10

A logical environment that organizes notebooks, clusters, jobs, repos, and permissions for a team or project.

Question 11

What is a Databricks cluster at a high level?

Answer 11

A set of compute resources (driver and workers) managed by Databricks, used to run Spark jobs, notebooks, and SQL queries.

Question 12

What is an all-purpose cluster?

Answer 12

A cluster intended for interactive use, such as notebooks and ad hoc development, often shared by multiple users.

Question 13

What is a job cluster?

Answer 13

A cluster that is created for a specific job or workflow run and typically terminates when the job completes, providing isolation and cost control.

Question 14

Why is the distinction between all-purpose and job clusters important?

Answer 14

It affects cost, reproducibility, isolation, and how you design dev vs production workflows.

Question 15

What is a notebook in Databricks?

Answer 15

An interactive environment for writing code (e.g., Python, SQL, Scala), running commands, and visualizing results, tied to a cluster.

Question 16

Why are notebooks popular for data engineering and ML work?

Answer 16

They support iterative exploration, visualization, and collaboration, while still allowing scheduling and productionization via jobs.

Question 17

What are Databricks Jobs?

Answer 17

Managed workflows that schedule and run notebooks, JARs, Python scripts, or multi-task DAGs on clusters, with retry and monitoring support.

Question 18

How do Databricks Jobs relate to external orchestrators like Airflow or Step Functions?

Answer 18

Jobs can be orchestrated by Databricks itself or triggered and managed from external orchestrators in larger workflows.

Question 19

What is a Databricks SQL warehouse (formerly SQL endpoint)?

Answer 19

A compute resource optimized for SQL workloads, allowing BI tools and users to run SQL queries against Delta tables via JDBC/ODBC.

Question 20

Why is Databricks SQL relevant for analytics teams?

Answer 20

It provides a warehouse-like experience on top of the lakehouse, enabling dashboards and ad hoc SQL using familiar BI tools.

Question 21

What is Delta Lake in the Databricks ecosystem?

Answer 21

An open table storage format that brings ACID transactions, schema evolution, and time travel to data stored on object storage.

Question 22

How does Delta Lake support the lakehouse idea?

Answer 22

By adding transactional semantics, reliability, and performance optimizations on top of raw data lake storage, allowing warehouse-like tables.

Question 23

What is the conceptual difference between a ‘bronze/silver/gold’ medallion architecture on Databricks?

Answer 23

Bronze holds raw ingested data, silver holds cleaned and standardized data, and gold holds curated, business-ready tables and aggregates.

Question 24

Why is the medallion architecture popular on Databricks?

Answer 24

It aligns well with Delta tables, Spark pipelines, and lakehouse principles, and makes lineage and quality boundaries explicit.

Question 25

What data engineering tasks is Databricks especially strong at?

Answer 25

Large-scale ETL/ELT, streaming ingestion and processing, complex transformations, and building unified batch/stream pipelines.

Question 26

What ML tasks are naturally supported by Databricks?

Answer 26

Feature engineering at scale, model training and tuning using Spark or single-node libraries, experiment tracking with MLflow, and batch scoring.

Question 27

What does Databricks add on top of raw Spark?

Answer 27

Managed clusters, optimized runtimes, Delta Lake integration, collaborative notebooks, job scheduling, MLflow, and governance features.

Question 28

Why is cluster lifecycle management a key benefit of Databricks?

Answer 28

It abstracts provisioning, scaling, and termination of Spark clusters, letting engineers focus on code instead of low-level infrastructure.

Question 29

How does Databricks help with performance compared to vanilla Spark?

Answer 29

Through optimized runtimes, auto-tuning features, Delta Lake optimizations, and cluster/configuration management.

Question 30

What is Unity Catalog at a conceptual level?

Answer 30

Databricks' governance and metadata layer for tables, views, and other assets, providing centralized access control and lineage.

Question 31

Why is centralized governance (Unity Catalog-style) important in a lakehouse?

Answer 31

It ensures consistent permissions, auditability, and lineage for data that might otherwise be spread across many object paths and clusters.

Question 32

How should you think about Databricks vs a traditional ETL tool?

Answer 32

Databricks is more of a programmable, Spark-based compute platform with notebooks and jobs, rather than a GUI-only pipeline designer.

Question 33

How should you think about Databricks vs a BI platform?

Answer 33

Databricks provides the data and compute engine; BI tools sit on top to visualize and explore data via SQL warehouses or connectors.

Question 34

Why is it helpful to separate dev, staging, and prod workspaces or environments in Databricks?

Answer 34

To test code and configuration safely before impacting production data and workloads, and to manage permissions and spend by environment.

Question 35

What roles typically interact with Databricks?

Answer 35

Data engineers, ML engineers/data scientists, analytics engineers, and platform engineers managing clusters and governance.

Question 36

Why is it important to keep Databricks projects repo-driven rather than notebook-only?

Answer 36

Repos and version control support CI/CD, code review, testing, and reproducible deployments, while notebooks alone can become brittle and ad hoc.

Question 37

What is the mental model for Databricks from a cost perspective?

Answer 37

You primarily pay for compute (clusters, SQL warehouses), while storage costs live in the underlying object store and can be optimized via Delta and lifecycle policies.

Question 38

In one sentence, what is the core mental model for Databricks in your stack?

Answer 38

Databricks is your programmable, managed Spark and Delta Lake engine on top of cloud object storage, where you do heavy ETL, streaming, and ML in a lakehouse architecture.