Stream processing

Question 1

some examples of streaming data (3)

Answer 1

1. 1. log files generated by customers using a mobile application
2. social network activity
3. e-commerce purchases

Question 2

what is stream processing

Answer 2

a processing mode where individual records or a small set of records are processed continuously, producing a simple response

Question 3

can streaming data be processed by batch processing?

Answer 3

yes

Question 4

what is bounded data?

Answer 4

datasets that are finite in size

Question 5

what is unbounded data?

Answer 5

datasets that are (at least theoretically) infinite in size and new data can arrive and be made available at any point of time

Question 6

What are streaming systems designed with in mind?

Answer 6

Unbounded data

Question 7

What is a data surge?

Answer 7

a sudden and significant increase in the volume of data flowing through a streaming data processing system.

Question 8

For real-time systems, why is failing to produce a processing result within a time window as bad as not producing
a result at all?

Answer 8

The events may become “insignificant” and the insights or trends produced may no longer be valid or accurate

Question 9

Examples of streaming data (4)

Answer 9

1. Messages from social platforms (e.g. Twitter)
2. Internet traffic going through a network device such as a switch
3. Readings from an IoT device
4. Interactions of users with a web application

Question 10

Frameworks for the ingestion of unbounded data (7)

Answer 10

1. Apache Kafka
2. Apache Flume
3. Amazon Kinesis Firehose
4. AWS IoT Events
5. Azure Event Hub
6. IoT Hub
7. Google Pub/Sub

Question 11

What are streams

Answer 11

sequences of immutable records that arrive at some point in time

Question 12

Other phrases for streams (3)

Answer 12

1. event streams
2. event logs
3. message queues

Question 13

What type of dataset are streams?

Answer 13

datasets in motion

Question 14

What type of dataset are tables?

Answer 14

datasets at rest

Question 15

what are the components of processing elements (PE)? (3)

Answer 15

1. input queue
2. computing element
3. output queue

Question 16

why are input queues needed? (3)

Answer 16

1. to maintain the order of the incoming events/data
2. to signal systems to slow down when the rate at which data is arriving might exceed the processing capacity of the stream processing system
3. to decouple the data source from the processing components allowing for more flexibility in management

Question 17

What is a spout in Apache Storm?

Answer 17

Elements that generate streams from external sources

Question 18

What is a stream in Apache Storm?

Answer 18

Unbounded stream of tuples

Question 19

What is a bolt in Apache Storm?

Answer 19

A processing element that consumes and generates streams

Question 20

What is a topology in Apache Storm?

Answer 20

a flow of spouts, streams and bolts

Question 21

what does the order of tuples in a stream represent?

Answer 21

the time at which they arrive at the streaming system

Question 22

what does the term at-least-once processing mean?

Answer 22

a guarantee that each message or event in a system will be processed at least once, but potentially more than once - ensuring that no data is lost in the event of failure

Question 23

what is event time?

Answer 23

the time at which one event has been generated by a source

Question 24

what is processing time?

Answer 24

the time at which events are seen by the stream processing system

Question 25

what are unordered streams?

Answer 25

streams where the event time ordering is different from the processing time ordering

Question 26

what is processing-time lag?

Answer 26

the delay between the generation and observation of an event in the system (this lag will change from tuple to tuple)

Question 27

when is event time or processing time relevant in processing? (2)

Answer 27

1. To aggregate tuples and produce an aggregate computation (e.g., count, average, ..) 2. To observe temporal patterns

Question 28

how do streaming systems to deal with temporal dimensions?

Answer 28

windowing

Question 29

what are the types of window?

Answer 29

1. fixed 2. session 3. sliding

Question 30

what does stream processing by windowing require? (2)

Answer 30

1. buffers to store tuples 2. a trigger stream that triggers the computation

Question 31

how does Spark streaming work? (4)

Answer 31

1. Receives input data streams 2. Divide the data into micro-batches by temporal windowing 3. Batches are treated as RDDs and processed by the Spark engine 4. Results are streamed as batches

Question 32

What does Spark streaming use for stream processing?

Answer 32

Micro-batches

Question 33

a key difference between streaming and micro-batch processing?

Answer 33

micro-batching has higher latency (delays) - up to the time interval defining the micro-batch

Question 34

why is the inverse reduction function used when using a sliding window?

Answer 34

it prevents the code from recomputing the entire aggregation from scratch for each window update, you use the previous aggregate value and apply an inverse operation to adjust for the outgoing and incoming elements

Question 35

What is Kafka described as?

Answer 35

“the HDFS of unbounded data sources”