Object storage (S3)
How Amazon S3 and similar stores work: keys and prefixes, erasure-coded durability, strong consistency, multipart uploads, presigned URLs, storage classes and pitfalls.
On this page
- The problem: petabytes of files and nowhere sensible to put them
- The model: buckets, keys and objects
- Durability: how it (almost) never loses data
- Consistency: strong since 2020
- Uploads and downloads
- Storage classes and lifecycle rules
- Worked example: the photo app
- Where it breaks
- When to use object storage, and when not
- In the interview
- Quiz
The problem: petabytes of files and nowhere sensible to put them
A photo app receives 10 million uploads a day, 2 MB each: 20 TB a day, about 7 PB a year. Putting them in the database bloats backups and replication. Putting them on the app servers' disks loses them when a server dies and makes every server a special snowflake. Running your own replicated file system means a team of storage engineers.
Object storage is the standard answer: a service that stores any number of files (objects) of almost any size under names (keys), replicates them across buildings, and serves them over HTTP. Amazon S3 (Simple Storage Service), launched in 2006, defined the model; Google Cloud Storage, Azure Blob Storage, Cloudflare R2, and self-hosted MinIO and Ceph offer the same idea, most with an S3-compatible API.
The object storage lesson introduces the basics: why files don't belong in the database, and blobs plus metadata. This lesson goes deeper into how S3 behaves, what it guarantees, what it costs and where teams get hurt.
The model: buckets, keys and objects
- A bucket is a container with a name that is unique across all of S3 (for ordinary "general purpose" buckets), living in one region.
- An object is the bytes plus metadata (content type, size, an
ETagchecksum, up to 2 KB of your own key-value metadata). Since December 2025 an object can be up to 50 TB (5 TB before). - A key is the object's full name, up to 1,024 bytes, such as
users/42/avatar.jpg.
Operations are simple HTTP verbs: PUT (create or replace the whole object), GET (read it, or a byte range of it), HEAD (metadata only), DELETE, and LIST (keys with a given prefix). There is no "edit byte 500", no append (except in the special Express One Zone class) and no rename: a rename is a copy plus a delete, and for a "folder" of a million objects, that is a million copies.
There are no folders, only keys that share a prefix
Durability: how it (almost) never loses data
S3 Standard is designed for 99.999999999% (11 nines) durability per object per year: store 10 million objects and you would expect to lose one every 10,000 years. It is designed for 99.99% availability (the SLA promises 99.9%). Durability and availability are different: an outage can make objects unreachable for an hour without losing any.
AWS does not publish the exact internal scheme, but it has described the ingredients: objects are split with erasure coding (cutting data into fragments plus computed parity fragments, so any large enough subset rebuilds it) and spread across at least three availability zones (separate data centres in a region), with background processes checking for and repairing lost fragments. The figure is a simplified picture of that.
Erasure-coded shards across zones, and a read that survives a zone failure
With 6 data and 3 parity fragments, an object takes 1.5× its size on disk instead of 3× for three full copies, and survives the loss of any 3 fragments. The price is that a read or a repair touches several machines.
Two points matter for designs. First, the metadata index (which key maps to which fragments) is a separate system from the bytes, which is why listing and the per-prefix request limits behave as they do. Second, eleven nines protect against hardware failure, not against you: a bug that deletes or overwrites objects is faithfully replicated. Versioning (keep old versions on overwrite and delete) and Object Lock (write-once retention) are the protection against that.
Consistency: strong since 2020
Until December 2020, S3 was eventually consistent for overwrites and deletes: right after replacing an object, a read could return the old one, and a new object might not appear in a LIST for a while. Many older articles, and old system design answers, still say this. It is no longer true.
S3 itself is read-after-write consistent; a CDN cache in front is not
Today S3 gives strong read-after-write consistency for PUT, GET, HEAD and LIST on all objects, in every region, at no extra cost. What it still does not give you:
- Locking or transactions. Two clients writing the same key at the same time: the last write to finish wins. Since 2024, conditional writes let you do better:
If-None-Match: *creates the object only if the key does not exist, andIf-Match: <ETag>replaces it only if it has not changed since you read it (a compare-and-swap). - Multi-object atomicity. Writing 100 objects is 100 independent operations; a crash leaves some written.
- Consistency of copies. Cross-region replication is asynchronous (usually seconds to minutes), and CDN or application caches keep serving what they cached.
GCS and Azure Blob are also strongly consistent; some S3-compatible stores are not, so check before relying on it.
Uploads and downloads
Multipart upload
A single PUT is limited to 5 GB, and one stream over a flaky mobile network is slow and restarts from zero on failure. Multipart upload splits a file into parts (5 MiB to 5 GiB each, up to 10,000 parts), uploads them in parallel and in any order, retries only the failed ones, and then calls complete to assemble the object atomically. Use it for anything above roughly 100 MB.
Large files go up in parts, in parallel, and resume after failures
Parts of an upload that is never completed or aborted are stored, and billed, indefinitely and invisibly (they do not show in LIST). Add a lifecycle rule AbortIncompleteMultipartUpload after a few days to every bucket.
Downloads work the same way in reverse: byte-range GETs in parallel fetch a large object several times faster than one stream, and let video players seek.
Presigned URLs
The app server should not carry gigabytes of video through itself. A presigned URL is a normal S3 URL with a signature in its query string, generated by your server with its credentials, granting one operation on one key until an expiry time (at most 7 days, and no longer than the signing credentials last).
import boto3
s3 = boto3.client("s3")
# Let this client upload one photo, directly to S3, within 10 minutes.
post = s3.generate_presigned_post(
Bucket="photos-prod",
Key=f"uploads/{user_id}/{upload_id}.jpg",
Conditions=[["content-length-range", 1, 20_000_000],
{"Content-Type": "image/jpeg"}],
Fields={"Content-Type": "image/jpeg"},
ExpiresIn=600,
)
# Send post["url"] and post["fields"] to the client; it POSTs the file there.
A presigned POST with a policy can enforce a size limit and content type; a presigned PUT is simpler but checks less. Anyone holding the URL can use it until it expires, so keep expiries short, use one key per upload, and do not log the URLs. After the upload, an S3 event notification (to SQS, SNS, Lambda or EventBridge, delivered at least once) tells your backend to scan, resize and record the object. See large files and media.
A CDN in front
Serve public or semi-public objects through a CDN (CloudFront, Cloudflare, Fastly) rather than straight from the bucket: edges are closer to users (tens of ms instead of 100+), transfer from S3 to CloudFront is free while internet egress from S3 costs roughly $0.09/GB, and the bucket can stay private, readable only by the CDN (CloudFront's origin access control). For private content, sign the CDN URLs or cookies instead of the S3 URLs.
Storage classes and lifecycle rules
Prices below are approximate, for us-east-1; check the current price list for real numbers.
| Class | Storage per GB-month | Access | Minimum duration | Use for |
|---|---|---|---|---|
| Standard | ~$0.023 | Milliseconds | None | Hot data |
| Intelligent-Tiering | Standard price, dropping as it cools | Milliseconds (archive tiers optional) | None; small monitoring fee per object | Unknown or changing access patterns |
| Standard-IA / One Zone-IA | ~$0.0125 / ~$0.01 | Milliseconds, plus a per-GB retrieval fee | 30 days; 128 KB minimum billed size | Accessed monthly or less; One Zone-IA only for re-creatable data |
| Glacier Instant Retrieval | ~$0.004 | Milliseconds, higher retrieval fee | 90 days | Archives read a few times a year |
| Glacier Flexible Retrieval | ~$0.0036 | Minutes to 12 hours (restore first) | 90 days | Backups |
| Glacier Deep Archive | ~$0.001 | 12–48 hours | 180 days | Compliance archives |
| Express One Zone | Higher | Single-digit ms, one zone | None | Hot data for analytics and ML jobs next to compute |
Lifecycle rules move or delete objects automatically by age and prefix: for example, move uploads/ to Standard-IA after 30 days, to Glacier Instant Retrieval after 180, and delete non-current versions after 90.
Worked example: the photo app
10 M uploads a day, 2 MB each, plus 4 resized versions totalling 0.5 MB; 200 M image views a day.
- Storage growth: 10 M × 2.5 MB = 25 TB/day, ~750 TB a month. In Standard that adds ~$17,000 to the monthly bill every month.
- Lifecycle: originals are rarely viewed after 30 days. Moving originals (80% of the bytes) to Glacier Instant Retrieval at ~$0.004 cuts their cost by about 80%; keep thumbnails in Standard for the CDN.
- Requests: 10 M uploads × 5 objects = 50 M
PUTs a day × ~$0.005 per 1,000 ≈ $250/day. That is small next to storage, but it shows why per-object costs matter. - Reads: 200 M views ≈ 2,300 a second on average. With a CDN hit rate of 95%, S3 sees ~115
GETs a second, well under the per-prefix limit.
Where it breaks
LIST pages 1,000 keys at a time
Other gotchas:
- Public buckets. Most data leaks from object storage are a bucket or object made public by mistake. Keep "Block Public Access" on and serve through a CDN.
- Many tiny objects. Per-request cost and latency (first byte in roughly 10–100 ms) dominate for millions of 1 KB objects; batch them into larger files or use a database.
- Treating it as a file system. Mounting a bucket (Mountpoint for S3, s3fs) works for reading big files sequentially, not for databases, locks or random writes.
When to use object storage, and when not
| Object storage | Block storage (EBS) | Network file system (EFS) | Database | |
|---|---|---|---|---|
| Good at | Unlimited files, cheap, durable, HTTP access, CDN origin | Low-latency random I/O for one machine | Shared POSIX files for many machines | Small records, queries, transactions |
| Weak at | Partial updates, renames, listing as a query, latency per request | Sharing, scaling beyond one volume | Cost per GB, throughput at scale | Large blobs |
| Use when | Media, backups, logs, data lake files | A database's disk | Legacy apps needing a shared directory | Metadata about the objects |
When not to use it: for data you update in small pieces (a database), anything needing sub-millisecond latency (a cache or local SSD), workloads that need file locks or renames of whole directories, and as your only index of what you stored.
In the interview
Users of a SaaS product can request an export of all their data: a ZIP of 1–20 GB built by a background job. They get an email with a download link, which should work for 7 days. Exports older than 30 days should not cost anything. Design the storage, the link and the cleanup.
Compare with our answer
The job builds the ZIP and writes it with multipart upload to exports/{tenant}/{export_id}.zip in a private bucket, then records it in the database. The email links to your app (/exports/{id}), which checks the user is logged in and allowed, then redirects to a presigned GET URL valid for a few minutes, so a forwarded email or leaked link is not a 7-day key to the data. A lifecycle rule on exports/ deletes objects after 30 days (and aborts incomplete multipart uploads after a day). Large downloads benefit from range requests so browsers can resume.
A strong answer mentions: multipart upload for large files · a private bucket · short-lived presigned URLs issued after an auth check · a database record of the export · lifecycle expiry instead of a cleanup job · aborting incomplete multipart uploads.
A team serves profile photos from https://bucket.s3.amazonaws.com/avatars/{user_id}.jpg through a CDN with a 1-day TTL; when users change their photo, the old one shows for a day. A nightly job lists all 80 million objects to compute each user's storage use. Uploads go through the app servers, which then PUT to S3. The bucket has versioning on and no lifecycle rules. What would you change?
Compare with our answer
A strong answer mentions:
- Overwriting
{user_id}.jpgbehind a CDN serves stale photos; use a new key per version (avatars/{user_id}/{hash}.jpg) stored in the database, so caches can keep objects forever. - Listing 80 M objects nightly is 80,000 sequential calls; record sizes in the database at upload time (from the event notification), or use S3 Inventory.
- Uploads through app servers waste bandwidth and CPU; use presigned POSTs with size and type limits.
- Versioning with overwrites keeps every old photo forever; add lifecycle rules to expire non-current versions and abort incomplete multipart uploads.
- The bucket should be private, readable only by the CDN, rather than referenced by its public S3 URL.
- Buckets hold objects under flat keys; "folders" are prefixes. Whole-object
PUT; no append, edit or rename. - 11 nines durability by design: erasure coding across at least three zones. Versioning and Object Lock protect against your own deletes.
- Strongly consistent (read-after-write, including
LIST) since December 2020. No locks: use conditional writes (If-None-Match,If-Match). - Single
PUTup to 5 GB; multipart (5 MiB–5 GiB parts, up to 10,000) for large files; objects up to 50 TB. - Presigned URLs/POSTs: direct client upload and download, short expiry, size limits via POST policy.
- CDN in front, bucket private, versioned keys for cacheability.
- Classes: Standard → IA (30-day minimum) → Glacier Instant (90) → Flexible → Deep Archive (180). Lifecycle rules automate it; small objects cost more than they save.
LIST= 1,000 keys per call, sequential: keep metadata in a database. ≥3,500 writes / 5,500 reads per second per prefix; back off on 503.
1. A user replaces their profile photo in S3 and the app immediately reads it back from S3. What does it get?
Since December 2020 every read after a successful write returns the new object. Stale photos in practice come from CDNs and other caches, not from S3.
2. A mobile app uploads 4 GB videos over unreliable networks, and many uploads fail near the end. What is the best change?
Multipart upload makes large uploads resumable and parallel. A longer timeout still restarts the whole 4 GB after any failure, and routing through servers adds a hop.
3. A nightly job lists all 60 million keys in a bucket to find files uploaded that day, and it now takes hours. What is the best fix?
LIST returns at most 1,000 keys per call, sequentially, and cannot filter by date. A database index of uploads answers the query directly; the page size cannot be raised.
4. An app gives users a presigned GET URL valid for 7 days in an email. What is the main risk?
A presigned URL is a bearer token. Link to your app instead, check permissions, and redirect to a URL valid for minutes. Signatures do not reveal the secret key.
5. A team moves a billion 10 KB thumbnails to Standard-IA to save money, and the bill goes up. Why?
Minimum billable size and per-object transition fees dominate for small objects. Standard-IA is cheaper per GB, but a 10 KB object is billed as 128 KB.
6. A bucket's sudden traffic spike to one prefix produces some 503 Slow Down errors. What should the client do?
Limits are per prefix, and S3 scales busy prefixes automatically over time. Backoff with jitter rides out the adjustment; the limit is not per bucket.
7. Two workers might create the same report file reports/2026-11-19.csv at once, and only the first should win. What does S3 offer?
Conditional writes (added in 2024) give create-if-absent. Object Lock prevents deleting or overwriting locked versions but does not stop a new version, and versioning keeps both.
Score 60% or more and the lesson is marked complete.