Design it yourself · Chapter 23

Design a video platform

Design a YouTube-style video platform from scratch: requirements, estimates, direct uploads, a transcoding pipeline, adaptive bitrate streaming through a CDN, and where it breaks.

IntermediateInterview priority: High40 min
On this page
  1. The problem: huge files, every kind of screen
  2. Step 1: pin down the requirements
  3. Step 2: quick estimates
  4. Step 3: split metadata from media
  5. Step 4: build the design
  6. Key decisions
  7. Where this breaks
  8. In the interview
  9. Where to go next
  10. Quiz

The problem: huge files, every kind of screen

Creators upload videos. Millions of people watch them on phones on a train, laptops on hotel Wi-Fi and 4K TVs on fibre. Each viewer expects the video to start in a second or two and keep playing, even when the connection gets worse.

Creators upload once; viewers everywhere watch

  1. Now a video platform, like YouTube. Creators upload videos; millions of people search, browse and watch them on phones, laptops and TVs, on fast and slow connections.

What makes this design interesting is the size of the data. A single upload can be bigger than every other piece of data a user will ever create, and watching it moves gigabytes. Most of the design is about keeping those bytes on the right path: never through your app servers, rarely from your own storage, mostly from caches near the viewer.

This is a first design. The full case study (linked at the end) goes deeper.

Step 1: pin down the requirements

Your turn: what would you ask?

List the functional and non-functional requirements you would agree on before designing.

Compare with our answer

A strong answer mentions: upload a video with title and description · watch with quality that adapts to the connection · search and browse · views and likes counts · start time under ~2 s and little rebuffering · uploads never lost · a processing delay after upload is acceptable · out of scope: live streaming, comments, recommendations, monetisation.

Functional

  • Creators upload videos (up to a few GB) with a title and description.
  • Viewers watch on any device; quality adjusts to their connection.
  • Viewers search and browse; videos show view and like counts.

Non-functional

  • Playback starts in under ~2 seconds, with rare rebuffering.
  • Uploaded videos are never lost (high durability).
  • A video may take minutes to become watchable after upload.
  • Watching must be highly available; view counts can be slightly behind.

Out of scope: live streaming, comments, recommendations, ads.

Step 2: quick estimates

Assume 50 million daily viewers watching 5 videos each, and 500,000 uploads a day averaging 10 minutes.

QuantityWorkingResult
Video starts250 M ÷ 86,400 s≈ 3,000 / s (peaks ~6,000)
Uploads500,000 ÷ 86,400 s≈ 6 / s
Storage per uploadoriginal (~1 GB) + about 5 renditions (~1 GB together)≈ 2 GB
New storage500,000 × 2 GB≈ 1 PB / day
Watch traffic250 M views × ~5 min × ~3 Mbit/s ≈ 110 MB each≈ 28 PB / day ≈ 2.6 Tbit/s average

The numbers tell the story. Uploads are rare (6 a second) but each is heavy. Watching is the giant: terabits per second, which no single datacentre should serve directly. Metadata (titles, counts) is tiny by comparison.

Step 3: split metadata from media

Small metadata in a database, huge media in object storage

  1. Split the problem in two. Metadata, meaning titles, likes and comments, is small and goes in a database. The media, the video files themselves, is huge and goes in object storage.

The first decision is the same as in every media-heavy design: metadata in a database, media in object storage.

TableKeyNotes
videosvideo_idowner, title, description, duration, status (uploading / processing / ready / failed), storage prefix
renditions(video_id, quality)240p … 4K, bitrate, codec, manifest key
view_countsvideo_idUpdated in batches, not per view (see below)
POST /v1/videos                {title, description, size}   → {video_id, upload_urls[]}
POST /v1/videos/{id}/complete  {part_etags[]}
GET  /v1/videos/{id}                                        → {title, counts, manifest_url}
GET  /v1/search?q=...&cursor=...

Step 4: build the design

The upload path

Uploads go straight to storage with presigned URLs, in resumable parts

The app server never carries video bytes. It creates the videos row with status = uploading and hands the client presigned URLs (signed, time-limited permission to write one object). The client uploads directly to a "raw" bucket using multipart upload: the file goes up in parts (say 16 MB each), several in parallel, and an interrupted upload resumes from the last finished part instead of starting a 2 GB file again.

Transcoding

Transcoding runs on workers fed by a queue

  1. Uploaded videos can’t be streamed as they are. They must be transcoded into many versions: 240p for weak connections, 1080p and 4K for big screens. That takes minutes per video, so it runs on workers fed by a queue.

The original file cannot be streamed as it is: it may be in any format, and one 4K file is useless on a phone with a weak signal. Transcoding converts it into a ladder of renditions (versions at different resolutions and bitrates), for example 240p at ~0.4 Mbit/s up to 1080p at ~5 Mbit/s and 4K at ~15–20 Mbit/s.

When the upload completes, an event puts a job on a queue. Transcoding is CPU-heavy (a rough rule: several CPU-minutes per minute of video for a full ladder), so 500,000 ten-minute videos a day keep thousands of cores busy. It is done as a pipeline:

Split into chunks, encode in parallel, then package

  1. Split the original into chunks of a few seconds.
  2. Encode every chunk into every rendition, in parallel across many workers. A 10-minute video finishes in about a minute instead of an hour, and a failed chunk is retried alone.
  3. Package the results as short segments (typically 2–6 s) plus a manifest: a small playlist file listing every rendition and its segments. The two standard formats are HLS (.m3u8 playlists) and MPEG-DASH (.mpd).
  4. Write renditions to storage, set status = ready, and index the metadata for search.

The watch path

Viewers stream from a nearby CDN, and quality adapts to their connection

  1. Viewers stream from a CDN, close to where they live. The player watches your connection and switches between versions as it speeds up or slows down: adaptive bitrate streaming. That’s why a video drops to blurry instead of stopping.

The player fetches the manifest, then requests segments one at a time from a CDN. After each segment it measures its download speed and how many seconds are buffered, and picks the next segment's quality: adaptive bitrate streaming (ABR). When the train enters a tunnel, quality drops for a few segments instead of the video stopping. Each segment is a normal, cacheable HTTP file, which is exactly what CDNs are good at.

At 2.6 Tbit/s, the CDN is not an optimisation, it is the design. With a 95% edge hit rate, only about 130 Gbit/s reaches the origin storage, and popular videos are almost never read from it.

The browse path, and the whole picture

Three paths: upload through a queue and workers, watch through a CDN, browse through app servers, a cache and search

  1. Finding videos needs a search index, fed from the metadata. Popular video pages are cached. Putting it together: an upload path through a queue and workers, a watch path through a CDN, and a browse path through app servers, a cache and search.

Video pages (title, counts, channel) are served by stateless app servers from the metadata database, with caching in Redis for popular videos. A search index is fed from metadata changes. Thumbnails come from object storage via the CDN, like everything else heavy.

PathVolumeComponents
Upload~6/s, GBs eachPresigned multipart upload → raw bucket → event → queue → transcoding workers → renditions bucket
Watch~3,000 starts/s, Tbit/s of segmentsPlayer → CDN edge → origin storage
Browsethousands of page views and searches/sLoad balancer → app servers → cache → metadata DB; search index

Key decisions

How many renditions, and when?

OptionHowGood atWeak at
Full ladder up frontEncode every rendition at uploadEvery video plays well everywhere immediatelyWasted compute and storage: most videos get few views
Basic first, more laterEncode a few common renditions (e.g. 360p, 720p) quickly; add 4K and better codecs when a video gets popularFast to publish; spend compute where views areTwo-stage pipeline; popular videos briefly lack top quality
On demand (just-in-time)Package or transcode when first requested, then cacheMinimal storage for the long tailFirst viewer waits; risky if many first views happen at once

Choice: basic first, more later. Views follow a long tail: a small share of videos gets most of the watching, so it pays to spend extra compute (higher resolutions, more efficient codecs like AV1, which save bandwidth but cost more to encode) only on those.

How is video delivered?

OptionGood atWeak at
One progressive MP4 fileSimple; any playerOne fixed quality: stalls on slow links, wastes bandwidth on small screens
ABR with HLS/DASH segmentsAdapts to the connection; segments cache perfectly in CDNs; seek by jumping to a segmentMany small files; a manifest to manage

Choice: ABR. It is the industry standard for on-demand video.

Counting views

Writing UPDATE videos SET views = views + 1 for 3,000 starts a second, all on a few viral rows, creates hot rows that lock and slow the database. Instead, players send view events to a log such as Kafka; a stream job deduplicates and aggregates them per video per minute, and writes one update per video per minute. Counts lag by a minute or so, which users never notice.

Where this breaks

In the interview

Your turn: the requirement changes

The platform now wants live streaming: a creator broadcasts and up to 1 million people watch, with at most ~10 seconds of delay. What changes, and what can you reuse?

Compare with our answer

Ingest changes: the creator streams continuously (usually RTMP or SRT) to an ingest server near them, which transcodes in real time into a ladder and cuts short segments (2 s or less) as they arrive. The manifest is a live playlist that updates every segment. Delivery is reused: the same HLS/DASH segments through the CDN with ABR; with 1 million viewers requesting the newest segment at the same moment, an origin shield and request coalescing are essential. There is no chunk-parallel transcoding of a finished file, so encoder capacity per stream must be reserved. Low-latency HLS or DASH with partial segments can get the delay down to a few seconds. The finished stream is saved as a normal video afterwards.

A strong answer mentions: a real-time ingest and transcoding step · short segments and a continuously updated manifest · reusing CDN delivery and ABR · origin shield and request coalescing for the synchronised spike · the latency trade-off of segment length.

Your turn: spot the bugs

A design: the mobile app uploads the video to the API server, which saves it to local disk and transcodes it into one 1080p MP4 in the request handler; viewers download that file from the API servers; each view runs UPDATE videos SET views = views + 1. List the problems.

Compare with our answer
  • Uploads through the API server: ties up connections and bandwidth for minutes; no resume. Use presigned multipart uploads to object storage.
  • Local disk: other servers can't see the file and a disk failure loses it. Store in object storage.
  • Transcoding in the request: takes minutes and blocks the server. Queue the job and use workers, chunk-parallel.
  • One 1080p file: stalls on slow connections. Produce an ABR ladder with HLS/DASH segments.
  • Serving from API servers: Tbit/s of traffic from one place. Serve through a CDN.
  • Per-view UPDATE: hot rows on viral videos. Aggregate view events through a stream.

A strong answer mentions: direct uploads to object storage · queue plus workers for transcoding · a rendition ladder with ABR · CDN delivery · aggregated view counts.

Where to go next

Video platform on one page
  • Metadata (title, status, counts) in a database; media in object storage. Never put video bytes through app servers.
  • Estimates: 250 M views/day → ~3,000 starts/s; ~110 MB per view → ~2.6 Tbit/s. 500 K uploads/day × ~2 GB ≈ 1 PB/day.
  • Upload: presigned URLs, multipart (resumable parts) → raw bucket → event → queue.
  • Transcode: split into chunks → encode each into a ladder (240p … 4K) in parallel → package HLS/DASH segments (2–6 s) + manifest → ready.
  • Watch: player reads the manifest, fetches segments from a CDN, picks quality per segment (ABR).
  • Spend compute on popular videos: basic renditions first, higher quality and better codecs later.
  • View counts: events → stream aggregation → batched updates, not a per-view UPDATE.
  • Breaks: viral cold start (origin shield, coalescing), poison uploads (retry limits, dead-letter queue), abandoned uploads (lifecycle rules).
Interactive labBuild it yourself on the design boardOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. Your platform serves 250 million views a day, each moving about 110 MB. Why is the CDN central to the design rather than an optimisation?

2. A viewer's connection drops from 20 Mbit/s to 2 Mbit/s mid-video. With adaptive bitrate streaming, what happens?

3. A 10-minute upload takes an hour to transcode on one worker. How do you make it available in minutes?

4. Most videos get very few views, while a small share gets most of the watching. How should that shape transcoding?

5. Viewers start 3,000 videos a second. Each start runs UPDATE videos SET views = views + 1. What goes wrong first?

6. A brand-new video goes viral and hundreds of CDN edges all miss and fetch the same segments from origin at once. What helps most?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.