Data Engineering Fundamentals

Question 1

What is the core goal of data engineering?

Answer 1

To design, build, and operate reliable systems that move, transform, and organize data so it can be queried and used by others, such as analytics, machine learning, and applications.

Question 2

How does data engineering differ from data science?

Answer 2

Data engineering focuses on infrastructure, pipelines, and reliability, while data science focuses on analysis, modeling, and decision-making using the data produced by those systems.

Question 3

How does data engineering differ from software engineering?

Answer 3

Software engineering centers on application logic and user-facing features, whereas data engineering centers on data flows, storage, schemas, and batch or stream processing, though both share coding and reliability practices.

Question 4

What are the three main planes in a modern data platform?

Answer 4

Ingestion, storage and modeling, and serving or consumption.

Question 5

What is a data pipeline at a high level?

Answer 5

A defined sequence of steps that takes data from one or more sources, transforms it, and writes it to one or more targets on a schedule or in real time.

Question 6

What is the difference between batch and streaming processing?

Answer 6

Batch processing works on finite chunks of data at intervals, whereas streaming processing handles data continuously as events arrive with low latency.

Question 7

What is a data lake in conceptual terms?

Answer 7

A central store for raw and lightly processed data, usually on cheap object storage and supporting many formats and use cases.

Question 8

What is a data warehouse in conceptual terms?

Answer 8

A structured store optimized for analytical SQL over curated, modeled data with well-defined schemas.

Question 9

What is a lakehouse architecture trying to achieve?

Answer 9

Combine the flexibility of a data lake with the reliability, performance, and governance of a data warehouse on a single storage layer.

Question 10

Why are file formats a core concern for data engineers?

Answer 10

They determine how efficiently data can be stored, read, partitioned, and processed by engines; poor choices increase cost and latency.

Question 11

What is the key difference between row-oriented and column-oriented storage?

Answer 11

Row storage stores all columns of a row together, while column storage stores each column separately across many rows.

Question 12

Why are columnar formats preferred for analytics workloads?

Answer 12

They allow column pruning, better compression, and efficient vectorized reads, which reduce I/O and speed up analytical queries.

Question 13

What is partitioning in the context of large datasets?

Answer 13

Splitting data into directory or logical segments based on one or more keys, such as date or region, so queries can skip irrelevant partitions.

Question 14

Why does partition selection matter for performance?

Answer 14

Good partition keys align with common query filters and yield balanced partitions, minimizing scanned data and skew.

Question 15

What is the small files problem in data lakes?

Answer 15

Having many tiny files increases metadata overhead and scheduling cost, making queries and jobs slower and less efficient.

Question 16

What is schema-on-read?

Answer 16

Storing raw or loosely structured data and applying a schema at query time when it is read by an engine.

Question 17

What is schema-on-write?

Answer 17

Enforcing structure and types at the time data is ingested so stored data conforms to a defined schema.

Question 18

Why is schema-on-read attractive for raw ingestion?

Answer 18

It allows fast onboarding of heterogeneous sources without blocking on full modeling, which is useful for exploration and rapid prototyping.

Question 19

Why is schema discipline still necessary even with schema-on-read?

Answer 19

Without agreed schemas for curated layers, queries become brittle, break easily, and data quality is hard to guarantee.

Question 20

What are the typical layers in a well-structured data platform?

Answer 20

Raw or landing, cleaned or staging, and curated layers such as marts or feature stores, sometimes with additional sandbox layers.

Question 21

Why separate raw, staging, and curated data?

Answer 21

To preserve original data, isolate cleaning and standardization logic, and keep curated layers stable and business-friendly.

Question 22

What is data modeling in the context of data engineering?

Answer 22

Designing how data is structured into tables and schemas, including relationships and grain, to support efficient and understandable querying.

Question 23

What is dimensional modeling?

Answer 23

A modeling approach using fact tables for events or measures and dimension tables for entities and context, optimized for analytical queries.

Question 24

What is the role of a fact table?

Answer 24

To store measurements or events with foreign keys to dimensions and numeric metrics that can be aggregated.

Question 25

What is the role of a dimension table?

Answer 25

To store descriptive attributes for entities such as customers, products, or dates, enabling slicing, filtering, and labeling of facts.

Question 26

What is denormalization and why is it common in analytics?

Answer 26

Denormalization stores redundant data to reduce joins, trading some storage and update complexity for simpler, faster analytic queries.

Question 27

What is orchestration in data engineering?

Answer 27

Coordinating the execution order, scheduling, dependencies, and error handling for multiple pipeline steps and jobs.

Question 28

Why is orchestration critical beyond just having individual scripts?

Answer 28

It ensures data arrives in the correct order, handles retries and failures systematically, and provides visibility into end-to-end workflows.

Question 29

What is idempotency for data pipelines?

Answer 29

The property that running a job multiple times with the same inputs produces the same final result without duplicates or inconsistencies.

Question 30

Why must ingestion and transformation jobs be idempotent?

Answer 30

Because failures and retries are inevitable and non-idempotent jobs can corrupt data or double-count events on reruns.

Question 31

What is data quality in a data engineering context?

Answer 31

The extent to which data is accurate, complete, timely, consistent, and valid for its intended use.

Question 32

Why should data quality checks be part of pipelines rather than manual ad hoc checks?

Answer 32

Automated checks catch issues early, prevent bad data from propagating, and make quality guarantees repeatable and observable.

Question 33

What is observability for data systems?

Answer 33

The ability to understand system behavior via logs, metrics, traces, and data-level checks such as freshness, row counts, and distributions.

Question 34

What is lineage in data platforms?

Answer 34

Traceable relationships showing how data moves and transforms from sources through intermediate steps to final outputs.

Question 35

Why is lineage important?

Answer 35

It helps debug issues, assess the impact of changes, and satisfy audit and compliance requirements.

Question 36

What is a data contract as a mental model?

Answer 36

A clear, versioned agreement about schema, semantics, and SLAs between data producers and consumers.

Question 37

Why do data contracts matter in larger organizations?

Answer 37

They reduce accidental breaking changes, clarify responsibilities, and improve trust between teams that share data.

Question 38

What is the main role of a data engineer in machine learning systems?

Answer 38

To deliver reliable feature and label pipelines with histories and monitoring so models can be trained and served correctly.

Question 39

What is the difference between offline and online data in ML?

Answer 39

Offline data is used for training and batch scoring, while online data is served in real time for low-latency predictions.

Question 40

Why is train–serve skew a data engineering concern?

Answer 40

Differences between offline and online feature pipelines often stem from data engineering choices, and skew can cause models to underperform in production.

Question 41

What are the main reliability risks in data engineering systems?

Answer 41

Late or missing data, schema changes, bad data quality, pipeline failures, and unanticipated changes in scale or usage.

Question 42

What are the main cost drivers in a modern data platform?

Answer 42

Storage volume and retention, compute used for batch and stream jobs and queries, and data transfer between systems or clouds.

Question 43

Why should a data engineer care about cost and not just performance?

Answer 43

Because data platforms can become very expensive, and good engineering balances reliability, performance, and cost constraints.

Question 44

What is a good one-sentence mental model for data engineering?

Answer 44

Data engineering is the discipline of turning messy, evolving source data into trustworthy, well-modeled datasets and APIs that can be safely consumed at scale.