Miscellaneous & Anti-Patterns

Question 1

What is a design pattern in data engineering?

Answer 1

A reusable solution to a common data problem, described at a high level so it can be adapted to specific technologies.

Question 2

What is an anti-pattern in data engineering?

Answer 2

A common but counterproductive approach that seems convenient initially but leads to reliability, performance, or maintainability problems over time.

Question 3

Why is recognizing patterns and anti-patterns valuable for a senior data engineer?

Answer 3

It helps choose proven approaches quickly, avoid known pitfalls, and reason about trade-offs in complex systems.

Question 4

What is the ‘raw → staging → curated’ layering pattern?

Answer 4

A pattern where raw data is ingested with minimal changes, staging standardizes and cleans it, and curated layers provide business-ready models.

Question 5

Why is mixing raw and curated data in the same tables or paths an anti-pattern?

Answer 5

It blurs semantics, increases the risk of using unclean data in production, and complicates lifecycle and governance.

Question 6

What is the ‘append-only logs’ pattern?

Answer 6

Storing events as immutable, time-ordered records and deriving state from them, which supports replay, auditability, and recovery.

Question 7

Why is in-place destructive updating of historical data generally an anti-pattern in analytics?

Answer 7

It destroys history, hinders debugging, and can break time-based analysis and reproducibility.

Question 8

What is the ‘write-ahead log plus derived views’ pattern?

Answer 8

Store raw events or transactions, and build derived summary tables and materialized views from them for fast queries.

Question 9

Why is ‘report directly from application OLTP databases’ usually an anti-pattern at scale?

Answer 9

It couples analytics to operational systems, risks performance impact on OLTP workloads, and provides inconsistent or incomplete data.

Question 10

What is the ‘single source of truth’ pattern?

Answer 10

Designating a canonical dataset or table as the authoritative version for a domain, with other copies derived from it.

Question 11

Why is letting many teams maintain their own uncoordinated copies of the same core data an anti-pattern?

Answer 11

It leads to divergent definitions, conflicting numbers, and complex reconciliation work across the organization.

Question 12

What is the ‘idempotent job’ pattern?

Answer 12

Jobs are written so rerunning them with the same inputs produces the same output, avoiding duplicates or corruption.

Question 13

Why are non-idempotent jobs an anti-pattern in scheduled and retried pipelines?

Answer 13

They can double-count records, leave partial updates, and require manual cleanup after failures.

Question 14

What is the ‘backfill-friendly’ pattern?

Answer 14

Designing pipelines so they can be safely run over historical ranges without special-case logic or manual intervention.

Question 15

Why is hard-coding dates or ranges in pipeline code an anti-pattern?

Answer 15

It makes backfills and parameterized runs difficult, forcing code changes for routine maintenance tasks.

Question 16

What is the ‘schema-first’ or contract-first pattern?

Answer 16

Defining schemas and contracts before building pipelines, using them as the source of truth for producers and consumers.

Question 17

Why is allowing schemas to evolve silently without communication an anti-pattern?

Answer 17

Unannounced changes break consumers, cause subtle bugs, and undermine trust in the data platform.

Question 18

What is the ‘slowly changing dimension (SCD) type 2’ pattern used for?

Answer 18

Capturing historical attribute changes in dimension tables so analyses can reflect what was known at different times.

Question 19

Why is rewriting historical dimension attributes without tracking changes an anti-pattern in many analytics systems?

Answer 19

It loses history and may produce misleading historical reports that reflect current attributes instead of past reality.

Question 20

What is the ‘late-arriving data handling’ pattern?

Answer 20

Designing pipelines and models to accept and correctly integrate records that arrive after their nominal processing window.

Question 21

Why is ignoring late-arriving data an anti-pattern for time-based metrics?

Answer 21

It can systematically undercount or misalign metrics, particularly in domains with delayed reporting or eventual consistency.

Question 22

What is the ‘feature store’ pattern in ML systems?

Answer 22

Centralizing feature definitions, storage, and serving so multiple models can reuse them consistently across training and inference.

Question 23

Why is embedding one-off feature code directly into each model script an anti-pattern?

Answer 23

It leads to duplicated logic, inconsistent definitions, and difficult maintenance when definitions or data sources change.

Question 24

What is the ‘small, composable jobs’ pattern?

Answer 24

Breaking pipelines into smaller steps that each do one thing well and can be tested, monitored, and reused independently.

Question 25

Why is the 'giant monolithic job' often an anti-pattern?

Answer 25

It is hard to understand, test, debug, and evolve; failures can require rerunning large amounts of work even for small changes.

Question 26

What is the 'orchestration as coordinator, not compute' pattern?

Answer 26

Using the orchestrator only to schedule and coordinate tasks, while actual data processing happens in specialized compute engines.

Question 27

Why is putting heavy data processing logic directly in the orchestrator an anti-pattern?

Answer 27

It entangles control logic with data logic, strains the orchestrator, and reduces portability and testability of transformations.

Question 28

What is the 'metric as a first-class object' pattern?

Answer 28

Defining metrics with clear formulas, dimensions, and ownership in a central semantic layer or catalog.

Question 29

Why is hard-coding metric logic separately in many reports and pipelines an anti-pattern?

Answer 29

It produces inconsistent numbers and makes changes to metric definitions complex and error-prone.

Question 30

What is the 'one fact, many slices' pattern in warehousing?

Answer 30

Designing fact tables at a clear grain so they can support many different analysis cuts via dimensions instead of many overlapping, redundant facts.

Question 31

Why is proliferating many similar fact tables with slightly different grains an anti-pattern?

Answer 31

It complicates modeling, confuses users, and makes reconciliation and maintenance more difficult.

Question 32

What is the 'observability baked in' pattern?

Answer 32

Including logging, metrics, and data checks in pipeline design from the start instead of adding them only after incidents.

Question 33

Why is treating observability as an afterthought an anti-pattern?

Answer 33

It delays detection of issues, increases time to resolve, and makes root-cause analysis much harder when problems occur.

Question 34

What is the 'cost-aware design' pattern?

Answer 34

Making architectural choices with explicit awareness of storage, compute, and transfer costs, and measuring impact over time.

Question 35

Why is ignoring cost until budgets are exceeded an anti-pattern?

Answer 35

By then, expensive patterns are deeply embedded, making optimization or re-architecture painful and disruptive.

Question 36

What is the 'testable pipeline' pattern?

Answer 36

Structuring transformations and code so they can be tested with small datasets and automated tests before running at full scale.

Question 37

Why is having pipeline logic that only runs in production with live data an anti-pattern?

Answer 37

Bugs are found late, on critical data, and often during business hours, causing avoidable incidents and manual debugging.

Question 38

What is the 'documented ownership' pattern?

Answer 38

Assigning each dataset, pipeline, and model clear owners responsible for quality, uptime, and evolution.

Question 39

Why is the 'everyone and no one owns it' state an anti-pattern?

Answer 39

Issues linger unresolved, quality degrades, and it is unclear who decides on schema or process changes.

Question 40

What is a good one-sentence mental model for patterns and anti-patterns in DE?

Answer 40

Prefer designs that are explicit, idempotent, testable, and reusable, and be suspicious of shortcuts that hide complexity, mix concerns, or make history and contracts opaque.