E10

Question 1

The main idea of the Lambda Architecture is…

Answer 1

To build Big Data systems as a series of layers

Each layer satisfies a subset of the desired properties and builds upon the functionality provided by the layers above it

Question 2

Batch Layer

Answer 2

1. Manages the master dataset – an immutable, append-only set of raw data
2. Pre-computes arbitrary query functions – called batch views

• Runs in a loop and continuously recomputes the batch views from scratch
• Very simple to use and understand
• Scales by adding new machines.

Question 3

Speed Layer

Answer 3

1. Accommodates all requests that are subject to low latency requirements
2. Does incremental computation instead of the recomputation done in the batch layer

Question 4

Speed Layer Goal

Answer 4

To have updated information on what happened since the last batch view was generated

Question 5

Serving Layer Goal

Answer 5

To merge views created by the batch layers with views created by the speed layer

Question 6

Difference Batch and Speed Layer

Answer 6

One big difference is that the speed layer only looks at recent data, whereas the batch layer looks at all the data at once

Question 7

Serving Layer

Answer 7

1. Indexes batch views so that they can be queried with low latency
2. The serving layer is a specialized distributed database that loads in a batch view and makes it possible to do random reads on it
3. When new batch views are available, the serving layer automatically swaps those in so that more up-to-date results are available
4. It does not need to support specific record updates
   - > This is a very important point, as random writes cause most of the complexity in databases

Question 8

Storing data in raw format has many advantages:

Answer 8

1. Data is always true (or correct): all records are always correct; no need to go back and re-write existing records; you can simply append new data
2. You can always go back to the data and perform queries you did not anticipate when building the system

Question 9

Data should be stored in raw format, should be

Answer 9

• Immutable

- Kept forever