Chapter 5 Designing Dropbox

Question 1

Requirements

Answer 1

1. Upload files from any device
2. Share with other users
3. Automatice sync between devices
4. Large Files
5. ACID compliant
6. Offline operations

Extended:
Snapshots

Question 2

Design considerations

Answer 2

1. Huge write and read volume
2. Read and write ratio is expected to be the same
3. Internally files can be stored as chunks of fixed size.
4. Reduce data traffic by updating chunks
5. Dedup the chunks
6. Keeping a copy of metadata with client can save a lot of round trips to the server
7. For small changes, client can just upload the diff.

Question 3

Capacity Estimation

Answer 3

1. 500M users, 100M daily active
2. Each user uses 3 different devices
3. about 100 billion files.
4. 10 PB of storage
5. 1 Million active connections per min

Question 4

High level design

Answer 4

1. Block server
2. Metadata server
3. Synchronization server

Question 5

Client design

Answer 5

1. Metadata saved in client as well.
2. Files are chunked based on average file size, cloud file block size, IOPS etc.
3. Only modified chunks are synced to backend.
4. Metadata changes are synced to sync server.
5. Updates from sync server are applied.
   6/ Internal metadata database.
6. Chunker.
7. Watcher
8. Indexer

Question 6

Metadata Database

Answer 6

1. Chunks
2. Files
3. User
4. Devices
5. Workspace

Question 7

Sync service

Answer 7

1. Metadata synchronization from clients.
2. Pushing updates to clients.
3. Message queue
4. Dedup blocks
5. Sync blocks

Question 8

Message queueing service

Answer 8

1. Request queue

2. Response Queue

Question 9

Storage service

Answer 9

Save chunks in cloud

Question 10

File processing workflow

Answer 10

1. A uploads chunks to cloud.
2. Updates metadata and commits changes
3. Notification are sent to B and C
4. B and C receive metadata changes and download chunks

Question 11

Deduplication

Answer 11

1. Post process deduplication

2. Inline deduplication

Question 12

Metadata partitioning

Answer 12

Use consistent hashing

Question 13

Caching

Answer 13

1. Hot file/chunks - Block storage cache

2. Metadata cache

Question 14

Load balancer

Answer 14

1. Between clients and block servers

2. Between clients and metadata servers.

Question 15

Security

Answer 15

1. Store metadata with permissions