Databricks Repos, CI/CD, and Dev Workflow

Question 1

What are Databricks Repos at a high level?

Answer 1

An integration that links a directory in the Databricks workspace to a Git repository, enabling version-controlled development inside Databricks.

Question 2

Why is using Repos preferable to ad hoc notebook copies for serious projects?

Answer 2

Repos keep code in Git with history, branching, and review, avoiding drift between local notebooks and source of truth.

Question 3

What kinds of files can be managed in a Databricks Repo?

Answer 3

Notebooks, Python modules, SQL files, configs, and other project artifacts that should be version controlled.

Question 4

How does a typical branch-based workflow look with Databricks Repos?

Answer 4

Developers work on feature branches in Repos, commit and push to Git, create merge requests, and sync changes into main when reviewed.

Question 5

Why should production jobs reference code from a Repo rather than ad hoc workspace notebooks?

Answer 5

So production code is versioned, reviewed, and reproducible, and changes flow through the same CI/CD pipeline as other software.

Question 6

What is the benefit of separating notebooks into ‘development’ and ‘production’ directories?

Answer 6

It discourages editing live production notebooks, keeps experimental work distinct, and clarifies which notebooks/jobs are stable.

Question 7

How can you structure a Databricks ML or DE project repo?

Answer 7

With folders for data prep, feature pipelines, training, evaluation, utilities, and job entrypoints, plus configuration files for environments.

Question 8

Why is it useful to centralize shared utilities in Python modules within the repo?

Answer 8

It avoids copy-pasting code across notebooks and allows reuse and testing of common logic.

Question 9

What is the role of configuration files (e.g., YAML/JSON) in a Databricks project?

Answer 9

They externalize environment-specific settings like paths, table names, and parameters so the same code can run in dev, staging, and prod.

Question 10

How does CI integrate with Databricks development?

Answer 10

CI pipelines can run unit tests, linting, and small integration tests against code in the repo whenever changes are pushed or merged.

Question 11

What tool can be used to run tests for Databricks code outside the workspace?

Answer 11

Standard Python testing frameworks like pytest, combined with mocks or local Spark sessions, and CLI tools or APIs to target Databricks when needed.

Question 12

Why is it valuable to have unit tests for transformation logic?

Answer 12

They catch regressions early, make refactoring safer, and ensure business logic is correct independently of the cluster.

Question 13

What is an integration test in the context of Databricks projects?

Answer 13

A test that runs end-to-end or multi-step flows on small datasets, often against a dev cluster or temporary Delta locations.

Question 14

How can Databricks Jobs be connected to CI/CD?

Answer 14

CI/CD pipelines can update job definitions (via Terraform or APIs) after successful tests and then trigger or validate test runs in dev/staging workspaces.

Question 15

What is the advantage of defining Databricks resources with infrastructure-as-code (IaC) tools?

Answer 15

IaC provides versioned, reviewable definitions for clusters, jobs, and permissions, enabling reproducible environments and controlled changes.

Question 16

Which IaC tools are commonly used with Databricks?

Answer 16

Terraform with the Databricks provider, and sometimes ARM templates/CloudFormation wrappers or custom scripts using the REST API.

Question 17

Why should promotion from dev to prod be automated rather than manual UI edits?

Answer 17

Automation reduces human error, ensures changes are tracked and repeatable, and makes it clear which code version is running where.

Question 18

What metadata is important to record for each deployment?

Answer 18

Git commit SHA, DBR version, cluster config, job definitions, and environment variables or config versions used in that release.

Question 19

How can feature flags be used in Databricks projects?

Answer 19

By controlling code paths with configuration or parameters, allowing gradual rollout or quick disabling of new logic without redeployment.

Question 20

What is a typical dev → stage → prod workflow on Databricks?

Answer 20

Develop and test on dev workspace, promote code and IaC changes to staging for realistic tests and backfills, then promote to prod once validated.

Question 21

Why is having separate Databricks workspaces/accounts for dev and prod beneficial?

Answer 21

It isolates experimental work from production data and jobs, reduces blast radius, and allows different access and cost controls.

Question 22

How can notebooks fit into a CI/CD pipeline?

Answer 22

By treating them as code files in a repo, testing underlying Python modules, and using job entrypoint notebooks that are deployed or referenced via IaC.

Question 23

What is a good practice for making notebooks more testable?

Answer 23

Minimize business logic in notebooks, moving most code into importable modules with tests, and let notebooks primarily orchestrate and visualize.

Question 24

Why is code review important for Databricks repos?

Answer 24

It helps catch errors, enforce standards, and share knowledge across data and ML engineers working on shared pipelines.

Question 25

What is the benefit of tagging releases in Git for Databricks projects?

Answer 25

Tags identify code snapshots tied to deployments, making it easier to reproduce or roll back to a specific version if needed.

Question 26

How do you ensure local development and Databricks environments stay consistent?

Answer 26

Use requirements files or environment specs, test code with the same library versions locally and in DBR, and avoid relying on ad hoc workspace-only installs.

Question 27

What is a runbook and how does it relate to CI/CD?

Answer 27

A documented procedure for handling deployments, failures, and rollbacks that complements automated pipelines with clear manual steps when needed.

Question 28

In one sentence, what is the core mental model for Repos, CI/CD, and dev workflow on Databricks?

Answer 28

Treat Databricks projects like any other software: keep code in Git, test and review it, define jobs and clusters as code, and promote changes through dev→stage→prod with automated, observable pipelines.