New Structured Streaming

Question 1

What is the concept behind streaming in big data (according to the PDF)?

Answer 1

Streaming processes data continuously as it arrives instead of waiting for full datasets. Structured Streaming extends the DataFrame API to handle data in real time

Question 2

When should you use streaming?

Answer 2

Use streaming when data arrives continuously (e.g., sensor feeds, logs, tweets, crime/incident data, weather, moving objects) and needs to be processed in near-real-time.

Question 3

What are the two main types of stream processing in Spark?

Answer 3

Micro-batching (Structured Streaming default) – processes ~1 MB batches at short intervals.

Continuous streaming – processes data one record at a time with lower latency

Question 4

How does micro-batching work?

Answer 4

Spark groups incoming data into small batches (e.g., every few milliseconds) and processes each batch like a small DataFrame job

Question 5

How does continuous streaming work?

Answer 5

Every record is processed immediately as it arrives, giving lower latency but requiring stricter guarantees.

Question 6

What are the output modes in Structured Streaming?

Answer 6

Complete: Outputs the entire result table every batch.

Append: Outputs only new rows
.
Update: Outputs newly added and updated rows.

Question 7

When is complete output mode typically used?

Answer 7

When debugging or when the entire output is small enough to print each batch.

Question 8

what is append output mode used for?

Answer 8

When the system should only output rows that have been newly added and won’t change

Question 9

What is update output mode?

Answer 9

Outputs both new rows and rows whose values have changed since the last batch.

Question 10

What are the main types of windows in streaming?

Answer 10

Tumbling windows
Sliding windows
Session windows

Question 11

What is a tumbling window?

Answer 11

A fixed-size, non-overlapping window (e.g., every 10 minutes)

Question 12

What is a sliding window?

Answer 12

A fixed-size window that advances by a slide interval (e.g., 10 min window sliding every 5 min). Windows overlap.

Question 13

What is a session window?

Answer 13

A window defined by periods of activity separated by gaps of inactivity. The gap is configurable or dynamic.

Question 14

What problem does watermarking solve?

Answer 14

Helps Spark handle late or out-of-order data by specifying how much lateness to tolerate before finalizing results.

Question 15

How is watermarking applied?

Answer 15

Developers call .withWatermark(“timestamp”, “10 minutes”) so Spark knows late data beyond 10 minutes can be ignored

Question 16

Why is watermarking needed in streaming systems?

Answer 16

Input data may not be totally ordered, especially when combining multiple ordered streams; watermarking prevents waiting indefinitely for late data