E4

Question 1

Some of the challenges of creating big data applications

Answer 1

1. Scaling problems
2. Fault-tolerance issues
3. Data corruption issues

Question 2

Best approach to scaling problems?

Answer 2

Use multiple database servers and spread the table across all servers.

Each server will have a subset of the data.

Question 3

Scaling using multiple databases. How?

Answer 3

1. Deploy more database servers
2. Use a different hash function
3. Redistribute the users according to the new hash function
4. Change the code of our application

Question 4

Fault-tolerance issues

Answer 4

When we have many databases it starts to become frequent that the hard drive in one of the databases goes bad

• We need to deal with having one of the databases down
• We need to add backups to each of the databases

Our system is not resilient to hardware errors

Question 5

Data corruption issues

Answer 5

At some point we deploy code with a bug: instead of incrementing each video viewership by one unit, our code increments by two units. We notice the mistake only 24 hours later.

Now we have corrupted data: every video watched in the past 24 hours have their viewership inflated. How do we solve this?

Our system is not resilient to human errors

Question 6

The desired properties of Big Data systems are related both to

Answer 6

Complexity and scalability

Question 7

Complexity

Answer 7

generally used to characterize something with many parts where those parts interact with each other in multiple ways

Question 8

Scalability

Answer 8

ability to maintain performance in the face of increasing data or load by adding resources to the system

Question 9

A big data system must

Answer 9

1. perform well
2. be resource-efficient
3. easy to reason about

Question 10

Desired properties of a Big Data system

Answer 10

1. Robustness and fault tolerance
2. Low latency
3. Minimal maintenance
4. Ad hoc queries