Infrastructure & Architecture

Question 1

scale-up (vertical scaling)

Answer 1

upgrade existing machine (SMP,MPP)

Question 2

scale-out (horizontal scaling)

Answer 2

adding more machines to network, unlimited scaling

Question 3

Symmetric Multi-processing (SMP)

Answer 3

traditional PC, multiple processors share same RAM and storage

Question 4

Massively Parallel Processing (MPP)

Answer 4

Each processor has its own dedicated RAM and storage (vendor lock-in)

Question 5

Cluster Architecture

Answer 5

many computers connected to work as a single system.

Question 6

how are nodes connected in cluster

Answer 6

usually gigabit ethernet, 8-64 per rack

Question 7

Only pro of MPP over cluster

Answer 7

faster message passing between nodes. Ideal for single, vertical solutions like data warehousing

Question 8

commodity hardware

Answer 8

standardized, market priced hardware

Question 9

distributed computing

Answer 9

tasks split into smaller units processed simultaneously across machines

Question 10

challenges of distributed computing (4) (ARFA)

Answer 10

Assigning SPLIT tasks,
Resource Allocation, Fault-tolerance, Aggregating (results)

Question 11

solution to challenges of distributed computing

Answer 11

Use a framework to hide complexity of distributed computing from developers

Question 12

Grid computing

Answer 12

• Collect computer resources from multiple locations.
• Each node perform different task
• Multi-purposed

Question 13

What is HPC?

Answer 13

high-performance computing (GPU intensive)

Question 14

What is NIST reference architecture for?

Answer 14

How a Big Data system should be designed

Question 15

5 Components of NIST Reference Architecture

Answer 15

1. Big Data Framework Provider
   - processing
   - storage
   - networking
2. Data Provider
3. Application provider
4. Data Consumer
5. System Orchestrator
   - integrate components
   - meet goals

Question 16

What happens in application component?

Answer 16

Programs are written to process data from collection to visualization and user access

Question 17

What happens in Data Provider component besides just data collection? (4) (Suck My Ass!)

Answer 17

1. Scrub sensitive info
2. Create Metadata
   - data provenance
   - access rights
   - usage policies
3. Enforce access and authorizations

Question 18

4 components professor adds to NIST

Answer 18

1. Analytical data store
2. Analysis and reporting
3. Real-time message
   ingestion (and buffer)
4. Stream processing

Question 19

Batch vs Stream

Answer 19

Running analytical algorithms on:

• Large amounts of
  stored data
• Continuously running,
  potentially infinite
  amount of data as soon
  as its collected

Question 20

Lamda Architecture

Answer 20

Two pipelines:
1. Cold path (batch)
2. Hot path (real-time)

Question 21

Kappa Architecture

Answer 21

One hot path (real-time)