File sync, like Dropbox or Google Drive
Chunking, deduplication, syncing across devices and resolving conflicts.
What is on this board
Parts
- Sync client Desktop app
- Web Web app
- Metadata Service
- Metadata DB Relational DB · MySQL
- Blocks Service
- Chunks Object storage · S3
- Change notifications WebSocket / realtime server
- Changes Message queue
- Scan Worker pool
- Two devices edited offline? Decision
- First write wins Option
- Merge Option
How they connect
- Sync client → Metadata: commit: these hashes
- Metadata → Metadata DB: new version
- Metadata → Sync client: missing: these two
- Sync client → Blocks: upload chunks
- Blocks → Chunks: save
- Metadata → Changes: changed
- Changes → Change notifications: tell other devices
- Change notifications → Web: pull the change
- Changes → Scan: scan
- Two devices edited offline? → First write wins: chosen
Notes and decisions
- Sync client: Splits files into 4 MB chunks, hashes each
- Metadata: Files, folders, versions
- Metadata DB: Sharded by account
- Chunks: Stored once per hash
- First write wins: The second becomes a conflicted copy. Nobody loses work. Chosen
- Merge: Impossible for most file types
- Only changed chunks travel, and the same chunk is stored once. Most uploads are a few bytes of metadata.
More like this
News feed, like TwitterFan-out on write or on read? Timelines in a cache, the celebrity problem, and how a post reaches millions of followers.
News feed, like TwitterFan-out on write or on read? Timelines in a cache, the celebrity problem, and how a post reaches millions of followers.
URL shortener, like BitlyThe first system design interview question: key generation, heavy reads, a cache in front, and click analytics off the request path.
