Structured Streaming

Question 1

Spark streaming

Answer 1

▪ Extends the core RDD API to support
streaming
▪ Not recommended

Question 2

Structured Streaming

Answer 2

▪ Extends the Dataframe API
(SparkSQL)
▪ More efficient and recommended
Streaming in Spark

Question 3

Output Mode: Complete

Answer 3

▪ Produce the entire output after each batch
▪ Usually useful for debugging and for small
output

Question 4

Output Mode: Append

Answer 4

▪ Only produce new output records
▪ No changes are allowed

Question 5

Output Mode: Produce

Answer 5

▪ Produce both new records and modified
records

Question 6

Timestamps

Answer 6

• In some cases, data might come with its
  own timestamps e.g ( Weather data,Crimes/Incidents)
• Structured Streaming allows developers to
  utilize these timestamps
• The assumption is that the data is already
  (partially) sorted by this timestamp

Question 7

Watermarking

Answer 7

• The input data might not be totally
  ordered
• For example, when merging multiple
  ordered streams
• Why not ensuring a total ordering?
• Watermarking is a Structured
  Streaming feature for informing Spark
  how much error to tolerate