Week 4: Integration & Processing Pipeline

Question 1

What is a data lake ?

Answer 1

• Holds a vast amount of raw data
• No hierarchy or organisation
• Holds structured, semi-structred, unstructred

Question 2

Data swamp

Answer 2

Highly disorganised data repository

Question 3

Data lake (RUDEAS)

Answer 3

1. Reason for storing data is undefined
2. Used by data scientist
3. Data is left raw until it is needed
4. Emerging technology
5. Adapt easily to changes
6. Schema on read

Question 4

Data Warehouse (DDRUSS)

Answer 4

1. Data is processed and ready to be queried
2. Difficult to change the structure
3. Reason for storing data is pre-defined
4. Used by business professionals
5. Strong maturity model
6. Schema on write

Question 5

3 Techniques of big data integration (DRS)

Answer 5

1. Data Fusion
2. Record Linkage
3. Schema Mapping

Question 6

Schema Mapping

Answer 6

Create a mediated global schema that is relevant to the business

Identfiy mappings between the schema and the data source

Question 7

Record Linkage

Answer 7

Identify records that refer to teh same logical entity across different data sources

Question 8

3 Record Linkage techniques (PCB)

Answer 8

1. Pairwise matching
2. Clustering
3. Blocking

Question 9

Data Fushion

Answer 9

A combination of techniques that aims to resolve conflicts from a collection of sources to find truth

Question 10

Data Fushion (techniques) (CSV)

Answer 10

• Copy detection
• Voting
• Source Quality

Question 11

Big Data Processing Pipeline 3 steps

Answer 11

1. Split
2. Do Something (machine learning)
3. Merge