Storage: DFS

Question 1

Distributed File System

Answer 1

An abstraction for data stored across multiple machines to appear as a unified storage system

Question 2

Goals of DFS

Answer 2

1. Split files
2. Hide complexity
3. Have Fault-tolerance

Question 3

Hadoop Distributed File System (4)

Answer 3

• Most widely used DFS
• Clusters of commodity
  hardware
• Handling up to petabytes of
  data
• Designed for high
  throughput batch (not low
  latency access)

Question 4

Blocks

Answer 4

Smallest unit of storage that can be read or written
- default size 64-128MB

Question 5

How does HDFS ensure data consistency?

Answer 5

Write-once, Read-many model

Question 6

Hadoop default Fault-tolerance

Answer 6

Blocks duplicated at factor of three to ensure data remains accessible if one machine fails (hardware failure = NORM!)

Question 7

Namenode (NN) (2)

Answer 7

Master in master-slave architecture
1. Store metadata about
location of specific blocks.
2. Control client access to data

Question 8

Datanodes (DN)

Answer 8

The slaves because they store and process the actual data. Send periodic “heartbeats” to update master

Question 9

The big issue with Master-Slave architecture

Answer 9

Single Point of Failure (SPOF), there is only one namenode that maintains the filesystem tree

Question 10

High Availability and how it’s achieved with Master-Slave Architecture

Answer 10

A system that can tolerate faults

Two separate machines as NNs:
- 1 Active State
- 1 Standby State

Question 11

Besides HA, another reason it’s beneficial to configure additional name nodes

Answer 11

of blocks in system is limited by RAM of NN, since they store metadata about blocks in memory

Question 12

Erasure Coding

Answer 12

A way to store less redundant data by splitting into smaller data cells called “Stripes”

with

Parity cells as backup pieces to help recover data.

If you lose some data cells, you can still rebuild thanks to parity cells. (Data cells are, of course split across nodes)

Question 13

Pros of erasure coding (2)

Answer 13

1. Reduce data redundancy
   from 200% to 50%
2. Faster writes without
   replicating

Question 14

Cons of Erasure Coding (3)

Answer 14

1. Higher cpu cost for reads
   and writes
2. Longer recovery time in
   case of failure
3. Loss of data locality (too
   much splitting!!)

Question 15

For which types of datasets does erasure coding work best?

Answer 15

Those with low I/O activities (not HOT or interactive)

Question 16

HDFS cons in general (2)

Answer 16

1. High latency
2. Not good if there
   are many small files
   limited by memory in NN.

Question 17

Standard data formats (3)

Answer 17

1. (Structured) CSV
2. (Semi-structured) JSON files 3. (Unstructured) textual files

Question 18

3 benefits of big data specific file formats (BCS)

Answer 18

1. Binary serialization
2. Compression
3. Splittability

Question 19

Row-oriented file format

Answer 19

Store data in rows so it’s easy to read or write full record of fields.

Question 20

Good application for Row oriented format

Answer 20

OLTP

Updating customers profile, adding a new order (access to all fields for the record)

Question 21

Column-oriented file format

Answer 21

store data by columns makes easier compression because same data type is stored in same record (type specific encodings)

Easier to access specific features of data for aggregation (reduced I/O for analytical queries)

Question 22

Good application for column oriented format

Answer 22

OLAP

Average salary of employees by department (get all instances for one field with one query!)

Question 23

Row-oriented file formats examples (3)

Answer 23

SequenceFiles (key-value pairs, designed for MapReduce)
Apache Thrift (good communication between programs)
Apache Avro (schema evolution)

Question 24

Column-oriented file formats examples (2)

Answer 24

ORC (first on Hadoop, designed for Hive - sql on hadoop)
Apache Parquet (schema evolution)

Question 25

Parquet

Answer 25

Columnar storage format for efficient querying of nested structures in a flat format