Big Data Management

Question 1

Characterization (3Vs)

Answer 1

Variety: Different forms of data
Volume: Petabytes of data
Velocity: Real-time data

Question 2

Big Data Analysis Pipeline

Answer 2

1. Data acquisition: Select important data to be stored
2. Information extraction & cleaning: Pull out required information from underlying sources
3. Data integration & aggregation & representation: Full integration not always possible. Problem, that origin can not be tracked on derived data, selection of storage complex
4. Modeling & analysis: Big data discloses hidden patterns and knowledge. Big pictures shows simple models
5. Interpretation: Annotate base data and discuss interpretation of metadata

Question 3

Data Lake requirements

Answer 3

1. Secure
2. Scalable
3. Reliable
4. Throughput
5. Low Latency
6. Store details
7. Store native Forman
8. All sources

Question 4

Advantages Cloud

Answer 4

• Cost
• Extensibility
• Reliability
• Workload
• Sharing

Question 5

Disadvantages Cloud

Answer 5

• Custom software
• Networking
• Maintenance
• Security
• Parallelization not always possible

Question 6

Three-Tier Server

Answer 6

Presentation ➔ Logic ➔ Data

Question 7

Design Cloud

Answer 7

• Transparent
• Flexible
• Reliable
• Performant
• Scalable

Question 8

Fallancies of cloud

Answer 8

1. Network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. Network is secure
5. Topology doesn’t change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

Question 9

Cloud characteristics

Answer 9

1. Dynamic
2. Massively scalable
3. Multi-tenant
4. Self-service
5. Per-usage based pricing model
6. IP-based architecture

Question 10

Google File System

Answer 10

Store chunks across chunk servers, replicate chunks, access control by master node

Question 11

Map Reduce

Answer 11

• Extract data as key value
• Group by key
• Reduce groups
• Split data and perform mapping parallel

Question 12

ACID

Answer 12

• Atomicity
• Consistency
• Isolation
• Durability

Question 13

CAP

Answer 13

• Consistency
• Availability
• Partition-tolerance
  ➔ Not all three possible at the same time

Question 14

BASE

Answer 14

• Basically Available
• Soft state
• Eventual Consistency

Question 15

Types of NoSQL storage

Answer 15

• Key/Value
• Wide-column
• Document database
• Graph database

Question 16

Steps of machine learning

Answer 16

Data ➔ Preprocessing ➔ Featuring ➔ Learning ➔ Testing ➔ Analysis

Question 17

Decision tree

Answer 17

• Created using greedy top down

- Choose attribute with most information value for each node step

Question 18

K-means clustering

Answer 18

• Variance within clusters minimal

- Random start points, assign data based on least distance to start points, recalculate start points, iterate