Streaming

Question 1

Data streaming

Answer 1

processing continuously generated data in near-real time so organizations can react instantly to changes

Question 2

2 reasons data streaming is important in modern big data systems

Answer 2

1. Businesses need faster access to insights
2. Processing workload is spread more evenly over time

Question 3

5 common use cases

Answer 3

1. Operational monitoring (temperatures, fan speeds in data center)
2. User activity on websites for fast personalized content
3. Automatically react to social media
4. IoT generates constant streams of data
5. Real-time bidding amongst advertising agents to decide which ad to show

Question 4

Approximations

Answer 4

Only a portion of dataset can be kept in memory, so sometimes data has to be approximated if its too much

Question 5

3 types of Data Streaming Models

Answer 5

1. Time Series Model (state updates)
2. Cash Register Model (increment state)
3. Turnstile (state increase or decrease)

Question 6

4 main layers of streaming architecture (CADM)

Answer 6

Collection tier
Analysis tier
Data Access tier
<–message queuing–>

Question 7

2 other optional layers of streaming architecture

Answer 7

In-memory storage (supports analysis)
long-term storage (keep for batch in future)

Question 8

What is collection tier made of?

Answer 8

Multiple edge servers that receive data from external sources (Normally TCP/IP-based over HTTP protocol, a lot of JSON format)

Question 9

Producer-broker-consumer concept

Answer 9

Producer is collection tier
Broker is messaging queue (many across nodes)
Consumer is Analysis tier

Question 10

3 main types of message delivery semantics

Answer 10

1. Exactly once (message delivered and processed once)
2. At most once (maybe message lost, but never processed more than once)
3. At least once (some duplicates occur, none lost)

Question 11

Analysis tier

Answer 11

Heart of the streaming architecture that adopts continuous query model and design algorithms specific to streaming problem

Question 12

Continuous query model

Answer 12

query that is issued once then continuously executed against the data, often maintaining a state. (Results regularly pushed to client)

Security system - query filters sensor data for human movement

Question 13

Windowing

Answer 13

Carry out analyses on a per-window basis instead of a simple per-item basis

Question 14

Sliding windows

Answer 14

define interval of analyses based off time

Question 15

Fixed windows

Answer 15

analyze last 5 minutes of data every 5 minutes

Question 16

Overlapping windows

Answer 16

analyze last 5 minutes of data every 2 minutes

Question 17

Sampling windows

Answer 17

analyze last 2 minutes of data every 5 minutes

taking samples, not all of it!

Question 18

data-driven windows

Answer 18

process only when session is active, and x time after it ends

don’t know the lengths ahead of time

Question 19

Event time vs Stream time

Answer 19

Event time - when event actually occurs (gold standard of windowing)
stream time - when event enters the streaming system

Time difference is called a skew

Question 20

Watermark

Answer 20

Captures progress of event time completeness as processing time progresses

Question 21

Perfect watermark

Answer 21

guarantees no late data ever arrives

Question 22

Heuristic watermark

Answer 22

estimates progress based on information available about the input stream (faster)

Question 23

Allowed Lateness

Answer 23

policy for accepting late data, since it is common

ex: allows 5 minute late data, everything else discarded

note - higher tolerance, longer data must be buffered

Question 24

Accumulation strategy

Answer 24

defines how intermediate results must be aggregated

Question 25

Single-pass algorithms

Answer 25

once examined, items discarded ex: count number of elements

Question 26

Approximated algorithms

Answer 26

Sampling and random projections for when one-pass algorithms don't work

Question 27

3 options for data storage after its been analyzed

Answer 27

discard it, push back to streaming pipeline, save for future use

Question 28

Data access tier

Answer 28

exposes analyzed data to consumer

Question 29

4 common protocols for data collection and data access tiers

Answer 29

webhooks, HTTP, Server-sent events (SSE), websockets