Streaming & Event-Driven Systems

Question 1

What is streaming data in the context of data engineering?

Answer 1

A continuous flow of records generated over time, often as events such as clicks, sensor readings, or log entries.

Question 2

How does streaming processing differ from batch processing?

Answer 2

Streaming processes data continuously as it arrives with low latency, while batch processes finite chunks at scheduled intervals.

Question 3

What is an event in an event-driven system?

Answer 3

A record representing something that happened at a specific time, often including a key, timestamp, and payload.

Question 4

What is a message broker or log-based streaming system?

Answer 4

A system that accepts, stores, and delivers ordered streams of messages or events to consumers, often partitioned for scalability.

Question 5

What is a topic in a streaming system?

Answer 5

A named stream or category of messages that producers write to and consumers read from.

Question 6

What is a partition in a topic?

Answer 6

An ordered, append-only sequence of messages that forms a shard of a topic for parallelism and scaling.

Question 7

Why are partitions used in streaming systems?

Answer 7

To distribute load across brokers and consumers and enable parallel reads and writes for scalability.

Question 8

What is an offset in a partition?

Answer 8

A monotonically increasing position that uniquely identifies a message’s location within a partition.

Question 9

What is a consumer group?

Answer 9

A set of consumers that coordinate to share the partitions of a topic so that each partition is consumed by only one member at a time.

Question 10

Why are consumer groups useful?

Answer 10

They allow horizontal scaling of consumption while providing a way to process each message once per group.

Question 11

What is at-most-once delivery semantics?

Answer 11

Messages are delivered zero or one time; they are never delivered more than once but may be lost.

Question 12

What is at-least-once delivery semantics?

Answer 12

Messages are delivered one or more times; they are not lost but may be processed more than once.

Question 13

What is exactly-once processing semantics (conceptually)?

Answer 13

Guaranteeing that each message’s effect is applied logically once, even if the underlying system uses retries or duplicates.

Question 14

Why is exactly-once processing hard to achieve in practice?

Answer 14

It often requires coordination across storage, compute, and sinks, idempotent operations, or transactional guarantees across systems.

Question 15

Why is at-least-once delivery commonly used in streaming pipelines?

Answer 15

It prioritizes durability and correctness, accepting duplicates that can be handled with idempotent logic or deduplication.

Question 16

What is idempotent processing in streaming?

Answer 16

Designing consumers so that processing the same message multiple times has the same effect as processing it once.

Question 17

What techniques help implement idempotent processing?

Answer 17

Using unique event IDs and upserts, tracking processed offsets or IDs, and designing sinks to ignore duplicates based on keys.

Question 18

What is event time?

Answer 18

The time at which an event actually occurred, as recorded in the event payload.

Question 19

What is processing time?

Answer 19

The time at which an event is processed by the system, which may lag behind event time due to delays or reordering.

Question 20

Why is distinguishing event time and processing time important?

Answer 20

Because analysis and windows should usually be based on when events happened, not when they were processed, especially with late data.

Question 21

What is late-arriving data in streaming systems?

Answer 21

Events that arrive after the system has already processed or closed the time window corresponding to their event time.

Question 22

What is a watermark in stream processing?

Answer 22

A marker indicating that the system believes it has seen all events up to a certain event-time, used to decide when to close windows.

Question 23

Why are watermarks used?

Answer 23

To balance waiting for late data against providing timely results by defining when windows can be considered complete.

Question 24

What is a window in streaming analytics?

Answer 24

A finite time range over which events are grouped for aggregation, such as 5-minute or hourly windows.

Question 25

What are tumbling windows?

Answer 25

Non-overlapping, fixed-size windows that partition the timeline, such as consecutive 1-minute intervals.

Question 26

What are sliding windows?

Answer 26

Windows of fixed size that advance by smaller steps, causing overlap, such as a 5-minute window that slides every 1 minute.

Question 27

What are session windows?

Answer 27

Windows defined by periods of activity separated by gaps of inactivity, used to group events into sessions rather than fixed intervals.

Question 28

What is stateful streaming processing?

Answer 28

Processing that keeps and updates state across events, such as counts, aggregates, or keyed state for joins and windows.

Question 29

What is a state store in streaming architectures?

Answer 29

A storage layer (in-memory, local disk, or external) used by stream processors to keep per-key or per-window state.

Question 30

Why does state management complicate streaming applications?

Answer 30

State must be checkpointed, recovered on failure, and scaled across nodes without losing consistency or performance.

Question 31

What is backpressure in streaming systems?

Answer 31

A condition where downstream operators cannot keep up with incoming data rates, causing queues to grow and slowing upstream producers.

Question 32

How can streaming systems handle backpressure?

Answer 32

By throttling producers, buffering with limits, scaling out consumers, or applying load shedding in extreme cases.

Question 33

What is a dead-letter queue (DLQ) in streaming pipelines?

Answer 33

A separate stream where messages that cannot be processed successfully after retries are sent for later inspection and remediation.

Question 34

Why are DLQs useful?

Answer 34

They prevent problematic messages from blocking the main stream and allow targeted manual or automated handling of bad records.

Question 35

What is stream–table duality?

Answer 35

The idea that a changelog stream can be viewed as a table evolving over time, and a table can be viewed as the result of accumulating a stream.

Question 36

What is a stream–stream join?

Answer 36

Joining two continuous streams of events, typically using time windows and keys to correlate events that occur within a time range.

Question 37

What is a stream–table join?

Answer 37

Joining a stream of events to a lookup table or state representing the latest values for keys, often used to enrich events with reference data.

Question 38

Why are ordering guarantees important in streaming?

Answer 38

Many computations, such as aggregations and state updates, assume events for a given key arrive in order; out-of-order data complicates logic.

Question 39

How is ordering typically defined in partitioned topics?

Answer 39

Ordering is guaranteed only within a partition, not across partitions; events may arrive out-of-order by event time even within a partition.

Question 40

What is a common partitioning strategy for topics?

Answer 40

Partitioning by a key such as customer ID or device ID so that all events for a key go to the same partition.

Question 41

What is a trade-off when choosing a partition key?

Answer 41

Keys that are too skewed create hot partitions; keys that are too random may complicate aggregation and join patterns.

Question 42

What is micro-batching in streaming systems?

Answer 42

Processing data in small, time-bounded batches approximating streaming behavior while using batch execution engines.

Question 43

What is end-to-end latency in streaming pipelines?

Answer 43

The time between when an event occurs and when its derived results become available to consumers.

Question 44

Why is monitoring lag important in streaming systems?

Answer 44

Lag indicates how far behind consumers are from the head of the stream, revealing performance issues or backlogs.

Question 45

What is a good one-sentence mental model for streaming data engineering?

Answer 45

Continuously consume ordered event streams, manage state and time carefully, and design idempotent, backpressure-aware pipelines that can handle out-of-order and late data.