Architecture · Chapter 18

Object storage

Where photos, videos and backups belong: block vs file vs object storage, how 11 nines of durability work, blobs plus metadata, presigned uploads, CDNs and storage tiers.

IntermediateInterview priority: High40 min
On this page
  1. The problem: files do not fit where rows live
  2. Three kinds of storage
  3. How it never loses your files
  4. Blobs plus metadata
  5. Uploading without carrying the bytes
  6. Serving downloads: CDNs and tiers
  7. When not to use object storage
  8. In the interview
  9. Quiz

The problem: files do not fit where rows live

A photo app lets users upload pictures. The first version stores each image as a BLOB column in Postgres, or as a file on the app server's disk. Both break quickly.

Big files don't belong in the database or on one server's disk

  1. Apps store more than rows of text. Photos, videos, PDFs and backups are big files: a single video can be bigger than your whole user table. Where should they live?
  2. Putting them in the database makes it huge, slow to back up and expensive. Keeping them on an app server’s disk is fragile: if that server dies, the files go with it, and other servers can’t see them.
  • In the database: a 2 MB photo is about 10,000 times bigger than the row that describes it. The database balloons, backups take hours, replicas lag while copying megabytes, and expensive database memory is wasted on bytes nobody queries.
  • On an app server's disk: only that server has the file. Add a second server behind the load balancer and half the requests cannot find it. If the disk dies, the photos die with it.

What you want is storage that every server can reach, that never loses files, that grows without limit, and that is cheap per gigabyte. That is object storage.

Three kinds of storage

KindWhat you getAccessed byGood atExamples
BlockA raw virtual disk; you put a filesystem or database on itOne machine at a time, mounted as a driveLow latency (sub-millisecond on SSD), small random reads and writesAWS EBS, a local NVMe SSD
FileA shared filesystem with folders and permissionsMany machines, through NFS or SMBShared files for tools that expect a normal filesystemAWS EFS, an NFS server
ObjectA flat store of whole objects under string keysAny client, through an HTTP API (PUT, GET, DELETE)Huge scale, high durability, low cost per GBAmazon S3, Google Cloud Storage, Azure Blob, MinIO

Databases sit on block storage because they rewrite small pages in place. User-uploaded media, backups, logs and data-lake files go in object storage.

Objects live in a bucket, each under a key

  1. Object storage, like Amazon S3 or Google Cloud Storage, is built for this. You put files, called objects, into a bucket, each under a name called its key, like photos/user42/beach.jpg.

An object store has buckets (top-level containers) holding objects, each under a key like photos/user42/beach.jpg. The slashes look like folders, but the namespace is flat: the key is just a string, and "folders" are only a shared prefix. You cannot edit a byte in the middle of an object; you upload a new version of the whole thing. In exchange you get practically unlimited capacity: one S3 object can be up to 5 TB, and a bucket can hold any number of them.

How it never loses your files

Every object is copied across machines and buildings

  1. It’s cheap, it scales to practically unlimited size, and it’s extremely durable: every object is copied across several machines and buildings, so losing one is astronomically unlikely.

Amazon S3 is designed for 99.999999999% (eleven nines) durability per object per year. In plain terms: store 10 million objects and you would expect to lose one about every 10,000 years. That comes from two techniques, plus constant checking.

Three full copies, or erasure coding at half the space

TechniqueHowStorage costSurvivesCatch
Replication (3 copies)Full copy on 3 disks in different zones3×Losing 2 copiesExpensive at petabyte scale
Erasure coding (e.g. 6 data + 3 parity)Split into fragments plus parity computed from them; any 6 of 9 rebuild the object1.5×Losing any 3 fragmentsRebuilding a fragment reads 6 others; more CPU and network on repair and reads

An availability zone is a separate datacentre (or group of them) with its own power and network. Spreading copies or fragments across zones means a fire in one building loses nothing. The system also scrubs continuously, re-reading data and comparing checksums, and repairs any fragment that has silently gone bad (bit rot) before a second failure can hit the same object.

Large stores use erasure coding for most data because at petabyte scale halving the disks is a huge saving; replication is still common for small or very hot objects. Note that durability is not availability: S3 Standard is designed for 99.99% availability, so a request can briefly fail even though the data is safe. Retry with backoff.

Blobs plus metadata

The database keeps the key; the bucket keeps the bytes

  1. Your database stores only the key, a tiny string. The app uses it to find the file when needed.

The pattern every design uses: the bytes (the "blob") go in object storage, and a small metadata row goes in your database.

CREATE TABLE photos (
  id           BIGINT PRIMARY KEY,
  owner_id     BIGINT NOT NULL,
  storage_key  TEXT NOT NULL,          -- photos/42/9f1c2e.jpg
  size_bytes   BIGINT,
  content_type TEXT,
  sha256       TEXT,
  status       TEXT NOT NULL,          -- pending | ready | deleted
  created_at   TIMESTAMPTZ NOT NULL
);

The database answers questions ("Ana's photos, newest first"); object storage serves the bytes. Use generated keys (an ID or content hash), not user-supplied filenames, so names never collide and nobody can guess other users' paths.

Uploading without carrying the bytes

A presigned URL lets the phone upload straight to the bucket

  1. For uploads, the app can hand the user a pre-signed URL: a temporary, permission-limited link. The phone uploads straight to the bucket, so your app servers never have to carry the heavy file.

A presigned URL is a normal storage URL with a signature attached: your server signs "PUT to this exact key, valid for 10 minutes" with its secret credentials, and gives the link to the client. The client uploads directly to object storage, and your app servers never touch the bytes. The full flow:

  1. Client asks the API to upload. The API inserts a photos row with status = pending and returns a presigned URL for a fresh key.
  2. Client PUTs the file straight to the bucket.
  3. The bucket emits an "object created" event. A worker checks the size and type, scans it, makes thumbnails and sets status = ready.

Limit what a presigned link allows: one key, a short expiry, and where the provider supports it, a size limit (S3's presigned POST policies can enforce a content-length-range). Always validate the file after upload as well.

Large files go up in parts, in parallel, and resume after failures

For big files, use multipart upload: the file is split into parts (S3 allows up to 10,000 parts of 5 MB to 5 GB each) uploaded in parallel and retried individually. A flaky phone connection at 95% then resumes from the last part instead of starting a 2 GB video again.

Serving downloads: CDNs and tiers

Lifecycle rules move cold data to cheaper storage; a CDN serves the hot data

  1. Files are often hot when new and rarely touched later. Lifecycle rules move old objects to cheaper cold storage automatically. And serving files through a CDN keeps downloads fast worldwide.

Serving every view from the bucket is slow for distant users and expensive: internet egress from cloud storage costs roughly $0.05–0.09 per GB. A CDN caches objects at edge servers near users. Name objects by content (9f1c2e.jpg) so they never change, and you can cache them for a year. For private files, the CDN can check signed URLs or signed cookies.

Most files are hot for days and then almost never read. Lifecycle rules move objects between storage classes automatically:

Class (S3 names)Rough price per GB-monthRead accessUse for
Standard~$0.023MillisecondsNew and popular media
Infrequent Access~$0.0125Milliseconds, plus a per-GB retrieval feeOld photos, last year's videos
Glacier Deep Archive~$0.001Hours to restoreCompliance archives, old backups

Worked example: a photo app's storage bill

1 million uploads a day × 2 MB = 2 TB a day, about 730 TB a year. Upload bandwidth averages 2 TB ÷ 86,400 s ≈ 23 MB/s, with peaks several times higher.

All in Standard after one year: 730,000 GB × $0.023 ≈ $17,000 a month, and growing. With a rule that moves photos older than 90 days to Infrequent Access, about 550 TB of that year is cheaper: 180 TB × $0.023 + 550 TB × $0.0125 ≈ $4,100 + $6,900 = $11,000 a month. Downloads matter as much: 50 TB of views a month at $0.09/GB from the bucket is $4,500, which a CDN with a high hit rate cuts substantially (CDN egress is cheaper per GB, and most requests never reach the bucket).

When not to use object storage

  • Small structured data you query or update in place: rows belong in a database.
  • Low-latency random writes: databases and anything rewriting small parts of a file need block storage. First-byte latency from object storage is typically tens of milliseconds.
  • Software that expects a filesystem (shared home directories, legacy tools that append to files): use file storage.
  • Tiny objects in huge numbers: billions of 100-byte objects cost more in per-request fees than in storage; batch them into larger files or keep them in a database.

In the interview

Your turn: design document storage for a law firm app

Lawyers upload contracts (1–50 MB PDFs), 200,000 a day. Documents are private, must be kept for 7 years, are read a lot in the first month and rarely after. Some must be impossible to delete before the 7 years end. Sketch the storage design.

Compare with our answer

Bytes in object storage under generated keys; a documents table with owner, matter, key, size, checksum, status and retention date. Uploads via short-lived presigned URLs (multipart for large files), validated and virus-scanned by a worker on the upload event. Downloads through presigned GET URLs that expire in minutes, checked against permissions in the app. Lifecycle: Standard for 30 days, Infrequent Access after, archive tier after a year. For "cannot be deleted", use the store's write-once retention lock (S3 Object Lock in compliance mode). Volume: 200,000 × ~10 MB ≈ 2 TB/day.

A strong answer mentions: bytes and metadata separated · presigned URLs for both upload and private download · validation after upload · lifecycle tiers matching the access pattern · a retention lock for the legal requirement · a rough daily volume.

Your turn: the requirement changes

Your photo app stores originals in object storage. Product now wants users to edit a photo (crop, filters) and keep the history of edits, and wants every photo to load in under 100 ms worldwide. What changes?

Compare with our answer

Objects are immutable, so each edit writes a new object under a new key (or uses bucket versioning); the metadata gets a photo_versions table pointing at each key, and the photo row points at the current version. Keep the original untouched. For speed: serve through a CDN with content-hashed keys so each version is cached forever and a new edit simply has a new URL (no purge needed); generate resized variants once per version. Old versions can move to a cheaper tier by lifecycle rule.

A strong answer mentions: objects are immutable, so edits create new objects · a versions table in the database · content-addressed keys so the CDN never serves a stale edit · pre-generated sizes · lifecycle rules for old versions.

Object storage on one page
  • Block = a disk for one machine (databases). File = a shared filesystem (NFS). Object = whole objects under keys over HTTP (media, backups, logs).
  • Bucket + key; flat namespace; objects are replaced, not edited. S3: up to 5 TB per object; strong read-after-write consistency since 2020.
  • Durability (eleven nines, design target) from replication (3×) or erasure coding (e.g. 6+3 = 1.5×, survives 3 losses), across zones, with scrubbing. Durability ≠ availability.
  • Bytes in the bucket, metadata (key, size, type, checksum, status) in the database.
  • Upload: presigned URL (one key, short expiry, size limit), multipart for big files, event → worker validates → ready.
  • Download: CDN with content-hashed keys and long TTLs; signed URLs or cookies for private files.
  • Lifecycle tiers: ~$0.023 → ~$0.0125 → ~$0.001 per GB-month. Egress costs roughly $0.05–0.09/GB.
  • Keep metadata and bytes in sync: pending status, sweepers, abort incomplete multipart uploads, reconciliation.
Interactive labPractise it: the storage playgroundOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. A team stores user videos (up to 2 GB) as BLOBs in Postgres. Which problem appears first as they grow?

2. Your mobile app uploads 500 MB videos through your API servers, which forward them to S3. Uploads fail often on weak connections and the servers' network is saturated. What should change?

3. Why do large object stores use erasure coding such as 6 data + 3 parity fragments for most data?

4. A user requests an upload URL, the app inserts a pending row, but the user closes the app before uploading. What is the best way to handle such cases?

5. Photos are viewed heavily for a week, then rarely. The bill for storage is growing fast. What helps most?

6. An avatar is served through a CDN with a one-year cache lifetime. Users who change their avatar still see the old one. What is the cleanest fix?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.