Media and real time · Design 22

Design Dropbox

Design Dropbox in a 45-minute interview: chunking and content-defined chunking, deduplication, metadata vs block storage, the sync protocol, conflicts and change notifications.

AdvancedInterview priority: High55 min
On this page
  1. How to use this page
  2. 1. Clarify the requirements
  3. 2. Back-of-the-envelope estimates
  4. 3. API
  5. 4. Data model
  6. 5. High-level design
  7. 6. Deep dives
  8. 7. Bottlenecks, failures and monitoring
  9. 8. Follow-up questions
  10. Quiz

How to use this page

This page follows the order of a real system design interview. At each step there is a Your turn box: write your own answer first (it stays in your browser), then open ours and compare. Reading a finished design feels easy; producing one under time pressure is the skill you are practising.

PhaseTime in a 45-min interviewWhat you produce
1. Clarify requirements5 minFunctional list, non-functional targets, what's out of scope
2. Estimate5 minStorage, upload bandwidth, number of blocks
3. API5 minThe upload, commit, change-feed and download calls
4. Data model5 minMetadata tables, the journal, the block index, where bytes live
5. High-level design10 minBoxes and arrows, and the upload and sync flows traced through them
6. Deep dives10–15 minChunking, sync, conflicts, notifications, durability
7. Wrap up2–3 minBottlenecks, failures, what you would monitor

Users have a folder that syncs across their devices. Files can be large. Changes on any device appear on the others. Users can share folders and restore old versions. Skip real-time co-editing (that’s Docs), search inside files, and the web preview of documents.

1. Clarify the requirements

Your turn: scope it

The interviewer says: "Design Dropbox." Write down what you would ask, then 4–6 functional requirements, 4–5 non-functional requirements with a target, and 2–3 things you would put out of scope.

Compare with our answer

The questions that matter most: How large can a file be? Do we need version history, and for how long? Shared folders? Must it work offline? Real-time co-editing, or whole-file sync? That last one changes everything: co-editing is the Google Docs problem (operations merged character by character); Dropbox syncs whole files and keeps conflicting copies side by side.

A strong answer mentions: upload/download any size · sync to every device · sharing · version history · offline edits that catch up · durability as the first non-functional requirement · co-editing out of scope.

Functional: what it must do

  • Upload and download files: Any size, from any device
  • Automatic sync: Changes appear on all devices
  • Share folders: With view or edit rights
  • Version history: Restore earlier versions for 30 days
  • Works offline: Sync catches up when back online

Out of scope

Real-time co-editingSearch inside filesDocument previews

Non-functional: how well it must work

QualityTargetWhy
DurabilityEleven nines: files must never be lostIt’s people’s only copy
Sync latencySeconds for small changes
Bandwidth efficiencyUpload only what changedLarge files, slow uplinks
ConsistencyEvery device converges; conflicts kept, never silently overwritten
Scale700 M users, exabytes stored

2. Back-of-the-envelope estimates

Your turn: the numbers

Assume 700 M users storing 20 GB each, deduplication saving 30%, 15% of users active per day, and each active user changing 50 MB a day. Work out total storage, upload bytes per day and per second, and the number of 4 MB blocks.

Compare with our answer
  • Raw storage: 700 M × 20 GB = 14 EB; after 30% dedup ≈ 10 EB. Physically more: erasure coding adds roughly 1.3–1.5× on top.
  • Active users: 700 M × 15% = 105 M. Changed data: 105 M × 50 MB ≈ 5 PB/day.
  • Bandwidth: 5 PB ÷ 86,400 s ≈ 60 GB/s on average (about 500 Gbps), more at peak.
  • Blocks: 10 EB ÷ 4 MB ≈ 2.5 trillion blocks, each needing an entry in the block index.
  • Metadata: a few hundred bytes per file version. Even 100 billion file versions is tens of TB: tiny next to the blocks, but read constantly.

A strong answer mentions: exabytes of blocks vs terabytes of metadata · bandwidth as the cost driver · the number of index entries · dedup saving · overhead of redundancy.

Assumptions: users 700 M; data per user 20 GB; saved by deduplication (%) 30; changed data per active user per day 50 MB; daily active share (%) 15.

NumberEstimateWhy it matters
Total stored (after dedup)9.8 EB≈ 10 EB: exabyte-scale block storage.
Uploaded per day5.2 PBWith chunking, only changed blocks travel.
Average upload bandwidth60.8 GB/s5 PB ÷ 86,400 s ≈ 60 GB/s.
4 MB blocks stored2.45 TEach needs a hash → location entry in the block index.
  • Metadata (file trees, versions) is tiny compared with blocks, but it’s queried constantly: keep it separate.
  • Many users store the same files (installers, popular PDFs): deduplicate by content hash.

The key insight: two kinds of data with completely different shapes. Blocks are exabytes, immutable and read rarely; metadata is small, mutable, transactional and queried thousands of times a second ("what changed in this folder?"). They must live in different systems. And at 60 GB/s of changes, sending only the changed parts of a file is not an optimisation, it is the budget.

3. API

Your turn: the API

A laptop edits a 2 GB file. List the calls it makes to upload the change, and the calls the phone makes to pick it up. Which call can fail with a conflict?

Compare with our answer

A strong answer mentions: ask which block hashes the server is missing · upload only those, addressed by hash · a separate commit that points the file at its new block list and carries the version it was based on · a change feed read from a cursor · a long poll (or push) that only says "something changed" · downloads by hash, through a CDN.

CallRequestResponseNotes
POST /v1/blocks/check{ hashes[] }{ missing[] }Which of these chunks does the server not have yet?
PUT /v1/blocks/{hash}4 MB chunk201Upload only the missing chunks
POST /v1/files/commit{ path, blockHashes[], baseVersion }{ fileId, version } or 409 conflictAtomically point the file at its new list of blocks
GET /v1/changes?cursor=–{ entries[], cursor, hasMore }Everything that changed since my cursor
GET /v1/notify?cursor=(long poll){ changes: true } after up to 60 sWakes the client when something changes
GET /v1/blocks/{hash}–chunk bytesDownload, often via CDN

Three things to point out about this API:

  • Blocks are addressed by their hash (PUT /v1/blocks/{sha256}). Uploading the same block twice is harmless: the second upload is a no-op. That makes block upload naturally idempotent (safe to retry), which matters on flaky networks.
  • Upload and commit are separate. The file only changes when the commit succeeds, and the commit is rejected unless every block it names is already stored. So the metadata can never point at a missing block.
  • baseVersion on commit is the hook for conflict detection: "I edited version 7". If the server is at version 8, the commit gets 409 Conflict.

4. Data model

Your turn: the tables

What do you store about a file, what do you store about a block, and how does a device find out what changed since it was last online? Pick a database for each.

Compare with our answer

A strong answer mentions: a file table keyed by namespace and path holding the version and ordered list of block hashes · a per-namespace append-only journal with a sequence number · a hash → location index with a reference count · blocks in object storage · sharding metadata by namespace so one folder's changes are in one place.

file metadata Sharded SQL (by namespace / user)

  • namespace_id, path, file_id
  • version, block_hashes[]
  • size, modified_at, deleted

File trees, versions and sharing are relational and need transactions; Dropbox ran this on sharded MySQL for years.

journal (changes) Sharded SQL, append-only per namespace

  • namespace_id, seq
  • file_id, op, version

An ordered list of every change; each device remembers its cursor in it.

block index Key-value store

  • sha256 → storage_location, refcount

hash → where the block lives; also tells us if we already have it (dedup).

blocks Object / block storage (Magic Pocket, S3)

  • blocks/{sha256}

Immutable 4 MB chunks, erasure-coded across data centres for durability.

Two terms in this model need defining:

  • A namespace is a tree of files that syncs as a unit: your personal Dropbox is one namespace, and each shared folder is its own namespace that is mounted into each member's tree. Sharding metadata by namespace keeps every change to one folder on one shard, so a commit and its journal entry fit in one local transaction.
  • The journal is an ordered, append-only list of every change in a namespace, numbered 1, 2, 3… A device remembers the last number it has applied: its cursor. "Sync" is then simply "give me journal entries after 5,812".

5. High-level design

Your turn: draw it

Sketch the boxes: where does a file's content go, where does its description go, and how does the phone learn that the laptop changed something? Then step through ours.

Compare with our answer

Compare your sketch with the figure. The pieces people most often forget: the split between metadata and blocks, the journal and cursor (instead of comparing whole folder trees), and a notification channel that carries no data.

The architecture, then editing one slide of a 2 GB file

  1. The key idea is separating metadata from blocks. A metadata service with a sharded database and a journal tracks files, versions and changes. A block service stores immutable chunks addressed by their hash. A notification service tells devices when to sync.
  2. You change one slide in a 2 GB file. The client splits the file into 4 MB chunks, hashes each, and asks which hashes the server lacks. Only the one or two changed chunks are uploaded, then the file’s new block list is committed.
ComponentWhat it does
Client sync engineWatches the folder, chunks and hashes files, keeps a local index
Metadata serviceFiles, folders, versions, sharing; commits are transactional
JournalOrdered change log per namespace; devices sync from a cursor
Block service + indexContent-addressed chunks; dedup by hash
Block storageErasure-coded, multi-region, durable
Notification serviceLong polls that wake devices when their namespaces change

Trace the upload out loud:

  1. The laptop's sync engine notices deck.pptx changed. It splits the file into chunks, hashes each with SHA-256, and compares them with its local index of the previous version: 2 of 500 chunks are new.
  2. It asks the block service which of those hashes the server lacks (/blocks/check). Both are missing.
  3. It uploads the two chunks (8 MB). The block service recomputes each hash to make sure the bytes match their name, then stores them.
  4. It commits: path = /deck.pptx, blocks = [500 hashes], baseVersion = 7. The metadata service checks the version, writes version 8 and appends a journal entry in one transaction.

Another device picks up the change

  1. The phone keeps a long-poll request open to the notification service. When the journal changes, it’s woken, asks for changes since its cursor, downloads only the chunks it doesn’t already have, and rebuilds the file.

The download side mirrors it: the phone is woken, reads journal entries after its cursor, sees the new block list, fetches only the hashes it doesn't have (here, two), and rebuilds the file locally. Then it saves its new cursor.

6. Deep dives

Your turn: the hard parts

Before reading on, write two or three sentences on each: how do you avoid re-uploading a 2 GB file for a small edit? What happens when two people edit the same file offline? How does a device learn about changes without polling all day?

Compare with our answer

A strong answer mentions:

  • Chunk and hash; upload only missing hashes; content-defined boundaries so inserts don't shift every chunk.
  • A commit that carries its base version; the loser of a race gets a conflict and saves a conflicted copy.
  • A journal per namespace with a cursor per device; a long poll that only says "changed".
  • Blocks first, then the metadata commit, so a crash can only leave unused blocks, never a broken file.

Chunking: why fixed-size chunks are not enough

Splitting files into 4 MB chunks named by their hash gives three things at once: delta sync (only changed chunks travel), deduplication (a chunk already stored, by you or anyone, is never stored twice) and resumable uploads (an interrupted upload keeps the chunks that finished).

Fixed-size chunks have one weakness: an insert. If you add a paragraph near the start of a file, every byte after it moves, so every 4 MB boundary falls on different bytes and every hash changes.

Fixed-size chunks against content-defined chunks

OptionHowCatch
Whole fileUpload the file on every change2 GB for a one-slide edit; minutes on a home uplink
Fixed-size chunksCut every 4 MB, hash eachInserts and deletes shift every later boundary
Content-defined chunksA rolling hash over a small window (about 48 bytes); cut where it matches a bit patternVariable chunk sizes; more CPU; needs min/max sizes
Binary diff (rsync-style)Send only the changed byte ranges against the old versionServer must hold or rebuild the old version; doesn't deduplicate

A rolling hash (Rabin fingerprint, or the faster Gear hash used by FastCDC) is a hash of the last few dozen bytes that can be updated in constant time as the window slides one byte. Cutting wherever its low 22 bits are zero gives chunks of about 2<sup>22</sup> bytes = 4 MB on average, with a minimum and maximum size to avoid tiny or huge chunks.

Choice: content-defined chunking for the file contents, plus per-chunk compression. Dropbox itself used fixed 4 MB blocks with an rsync-style diff inside changed blocks; both answers are defensible if you explain the insert problem.

Deduplication and its privacy problem

Because blocks are named by their content, a chunk that already exists anywhere in the store is never uploaded or stored again: the /blocks/check call returns "not missing". Popular installers, PDFs and photos shared across many accounts are stored once.

Metadata vs blocks: the decision that shapes everything

MetadataBlocks
SizeTens of TBAround 10 EB
ChangesConstantly (renames, new versions)Never: a block is immutable
NeedsTransactions, "what changed since?" queriesCheap, durable bytes
StoreSharded SQL (Dropbox used MySQL, later its own Edgestore on top)Object/block storage (Dropbox built Magic Pocket)

Immutability is what makes blocks easy: no locking, perfect caching in a CDN (a hash never changes meaning), and integrity checking for free (recompute the hash). All the hard concurrency lives in the small metadata store, where SQL transactions handle it well.

Order matters: upload blocks first, commit metadata second. A crash in between leaves orphan blocks (reference count 0) for the garbage collector; the reverse order could leave a file pointing at bytes that don't exist.

The sync protocol: journal and cursor

The naive way to sync is to compare the device's whole folder tree with the server's. With 100,000 files, that is a lot of work every time. Instead:

  1. Each commit appends an entry to the namespace's journal in the same transaction (so a committed change always has an entry, and entries are in commit order).
  2. Each device stores a cursor per namespace: the last journal sequence it applied.
  3. Sync is GET /changes?cursor=5812 → entries 5,813 onwards, a new cursor, and hasMore for paging.

This makes sync resumable: a phone that was offline for a week just reads a longer page. If the cursor is too old (the journal was compacted), the server returns "reset" and the device falls back to a full tree comparison once.

The three trees in the sync engine

Dropbox's rewritten sync engine (Nucleus, 2020) describes sync as three trees on the client: the remote tree (the server's latest state, from the journal), the local tree (what's on disk, from file-system events) and the synced tree (the last state both agreed on). Comparing local with synced tells you what the user changed; comparing remote with synced tells you what others changed. If both changed the same file differently, that is a conflict. Keeping the "merge base" explicitly is what lets the engine tell "I deleted it" from "I never had it", which naive two-way comparisons get wrong.

Conflicts: two people edit the same file offline

Two offline edits to the same file

  1. Two people edit the same file while offline. The first to sync wins the commit. The second commit is based on an old version, so the server returns a conflict and the client saves a “conflicted copy” instead of overwriting anyone’s work.
OptionHowCatch
Last writer winsAccept every commitSomeone's work silently disappears
Lock the file while editingCheck out / check inImpossible offline; forgotten locks block everyone
Merge automaticallyCombine the two versionsImpossible for arbitrary binary files
Optimistic concurrency + conflicted copyCommit carries baseVersion; on mismatch the client saves its version as "deck (Ben's conflicted copy).pptx"A human has to reconcile two files

The commit check is one conditional update: UPDATE files SET version = 8, blocks = ? WHERE namespace_id = ? AND path = ? AND version = 7. If no row matches, someone else won and the server returns 409. This is optimistic concurrency control: no locks, just a check at write time that the thing you read hasn't changed.

Change notifications: waking 100 M devices cheaply

Devices should hear about changes within seconds without polling. Each device keeps one long poll open (GET /notify?cursor=…): the server holds the request for up to about 60 seconds and answers early only if one of the device's namespaces moves past its cursor. The answer carries no file data, only "something changed"; the device then calls /changes. That separation keeps the notification service simple and stateless enough to hold tens of millions of idle requests.

Durability: eleven nines

Storing three full copies costs 3× the raw space. Erasure coding splits a block into, for example, 6 data fragments plus 3 parity fragments; any 6 of the 9 rebuild it. That survives 3 lost fragments at 1.5× overhead. Spread the fragments across machines, racks and zones, and scrub continuously: background jobs re-read blocks and recompute their hashes, so silent disk corruption is found and repaired before a second failure lands on the same block.

7. Bottlenecks, failures and monitoring

Scaling bottlenecks

BottleneckFix
Metadata hot spots (huge shared folders)Shard by namespace; cache directory listings; rate-limit sync storms
Exabytes of blocksOwn storage system (Magic Pocket) with erasure coding; cold blocks on dense disks
Notification fan-out for shared foldersNotify per namespace; devices pull changes themselves
First sync of a new deviceDownload from the CDN in parallel; LAN sync between devices on the same network

When things fail

FailureWhat happens
Upload interruptedChunks already uploaded are kept; resume with the rest
Commit fails after uploadOrphan blocks are garbage-collected later (refcount 0)
Metadata shard downSync pauses for those namespaces; local files keep working
Corrupted block detectedRebuilt from erasure-coded fragments in other zones

What to monitor

Sync latency (commit → other device)Upload/download throughputConflict rateBlock store durability checksMetadata DB latencyNotification fan-out lag

8. Follow-up questions

Your turn: the interviewer pushes

Answer these three in order before opening ours. 1) A 50 GB video upload fails at 80% on a hotel Wi-Fi. What happens when it resumes? 2) Version history must go from 30 days to 10 years for business customers. What changes? 3) A user renames a folder with 200,000 files. What does the journal look like?

Compare with our answer

A strong answer mentions:

  1. Chunks already uploaded are stored and found by /blocks/check; only the remaining 20% is sent; nothing is committed until all blocks exist.
  2. Old versions keep their blocks alive through reference counts, so storage grows; move old, rarely read blocks to colder, denser storage; GC runs on version expiry, per plan.
  3. Don't write 200,000 path changes: store a folder as an entry with an ID and children pointing at it by ID, so a rename is one metadata change and one journal entry; clients apply it locally.
How does deduplication interact with privacy?

Dedup by hash can reveal that someone stores a known file; many services dedup only within a user or encrypt per user.

How do you share a folder?

Shared folders are their own namespace; each member’s tree mounts it; the journal is per namespace.

How would you support LAN sync?

Devices on the same network discover each other and copy chunks directly, checking hashes, before falling back to the cloud.

Dropbox on one page
  • Numbers: ~10 EB of blocks after dedup (plus erasure-coding overhead), ~5 PB/day changed (~60 GB/s), ~2.5 trillion 4 MB blocks; metadata is only tens of TB.
  • Split: metadata (sharded SQL, transactional) vs blocks (immutable, content-addressed object storage).
  • Chunking: 4 MB chunks named by SHA-256; upload only missing hashes. Content-defined chunking (rolling hash) survives inserts.
  • Dedup: by hash; privacy risk across users, so prove possession or dedup per account.
  • Order: upload blocks → commit metadata. Crash leaves orphan blocks for GC, never a broken file.
  • Sync: per-namespace journal + per-device cursor; changes?cursor= is resumable.
  • Notify: long poll that says only "changed"; jitter for big shared folders.
  • Conflicts: commit with baseVersion; 409 → conflicted copy. Never silently overwrite.
  • Durability: erasure coding across zones (e.g. 6+3 at 1.5×), continuous hash scrubbing.
Mock interviewPractise it as a mock interview: scope, size and the big decisionsOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. A user inserts one paragraph near the start of a 500 MB log file. With fixed 4 MB chunks, roughly how much is uploaded?

2. Why does the client upload blocks before committing the file's new block list?

3. Two devices both edit version 7 of a file offline. The laptop syncs first. What should happen when the phone commits?

4. A phone has been offline for three days. How does it find out what changed?

5. A storage team proposes cross-user deduplication where the client sends only a hash and the server links the existing block. What is the main risk?

6. Which choice best explains keeping metadata and blocks in separate stores?

7. A shared folder with 10,000 members changes. What is the most likely immediate problem?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.