Design Notifications
Design a notification platform in a 45-minute interview: channels, templates, preferences, priority queues, rate limits, retries, idempotency and provider failover.
On this page
How to use this page
This page follows the order of a real system design interview. At each step there is a Your turn box: write your own answer first, then open ours and compare. "Design a notification system" sounds like "call Twilio in a loop". The interview is really about reliability at the edges: outside providers that fail and rate-limit you, callers that retry, users who opt out, and a login code that must never wait behind ten million marketing emails.
| Phase | Time in a 45-min interview | What you produce |
|---|---|---|
| 1. Clarify requirements | 5 min | Channels, who calls us, priorities, what's out of scope |
| 2. Estimate | 5 min | Sends/s and peaks, SMS cost, log volume |
| 3. API | 5 min | Send, campaign, status, preferences, device registration |
| 4. Data model | 5 min | Preferences, devices, templates, delivery log, dedupe keys |
| 5. High-level design | 10 min | API → queues by priority → workers → channel adapters → providers |
| 6. Deep dives | 10–15 min | Priorities, duplicates, provider failure, user protection |
| 7. Wrap up | 2–3 min | Bottlenecks, failures, monitoring |
Internal services ask us to notify a user by push, email or SMS using a template. Respect each user’s preferences and quiet hours, retry failures, avoid duplicates, and track delivery. Bulk campaigns to millions must not delay urgent messages. Skip building the campaign editor UI and an in-app inbox.
1. Clarify the requirements
The interviewer says: "Design a notification system for our company." Write the questions you'd ask, then 5–6 functional requirements, 4–6 non-functional requirements with targets, and what you'd leave out.
Compare with our answer
Useful questions: Which channels: push, email, SMS, in-app? Who sends: internal services only, or also marketing tools? Which messages are urgent? Do we own templates and preferences, or do callers? Do we need delivery tracking? Volumes and peaks?
A strong answer mentions: one API for several channels · templates with parameters · preferences, opt-outs and quiet hours enforced centrally · different latency targets for urgent and bulk · at-least-once with de-duplication · legal compliance for opt-outs.
Functional: what it must do
- Send via push, email or SMS: One API for all channels
- Templates: “Your order {id} has shipped” with parameters
- User preferences: Opt-outs per channel and topic, quiet hours
- Bulk campaigns: Millions of recipients
- Delivery tracking: Queued, sent, delivered, opened, failed
- Device registration: Push tokens per device
Out of scope
Campaign editor UIIn-app notification inboxA/B testing
Non-functional: how well it must work
| Quality | Target | Why |
|---|---|---|
| Latency (urgent) | OTP / security alerts in < 5 s | Users are waiting for the code |
| Latency (bulk) | Minutes is fine | Marketing can wait |
| Reliability | At-least-once, with no visible duplicates | Lost OTPs lock people out; duplicates annoy |
| Availability | 99.99% for the API | Every product depends on it |
| Scale | 1 B/day; campaign peaks of 10 M in minutes | |
| Compliance | Honour opt-outs and unsubscribe laws | Fines and app-store bans otherwise |
Three terms to fix early. A channel is a delivery route: push (through Apple's APNs or Google's FCM), email (through a provider such as SES or SendGrid) or SMS (through a gateway such as Twilio). A provider is the outside company that actually delivers on a channel; you don't run phone networks or mail servers yourself. A transactional message is triggered by something the user did (a login code, "your order shipped"); a marketing message is sent to a segment of users by a campaign.
2. Back-of-the-envelope estimates
Assume 1 B notifications a day, peaks 10× the average during campaigns, 2% sent by SMS at about $0.008 each, and 500 bytes of delivery log per notification. Work out the average and peak send rate, the daily SMS bill, and the log volume.
Compare with our answer
- Average: 1 B ÷ ~86,400 ≈ 11.6 K/s. Peak at 10×: ~116 K/s.
- SMS: 2% of 1 B = 20 M a day × $0.008 = $160 K a day, about $58 M a year. SMS is by far the most expensive channel, so prefer push and email wherever the user allows it.
- Logs: 1 B × 500 bytes = 500 GB a day. Keep detail for 30–90 days, then aggregates.
- Provider limits, not your servers, often set the real ceiling: an SMS sender number or an email domain has its own send rate.
A strong answer mentions: average and peak rates · SMS cost as a design driver · log volume and retention · provider rate limits as the true bottleneck · peaks caused by campaigns, not organic traffic.
Assumptions: notifications per day 1 B; peak vs average 10; share sent by sms (%) 2; cost per sms (us cents) 0.8; log record size 500 B.
| Number | Estimate | Why it matters |
|---|---|---|
| Notifications per second (avg) | 11.6 K / s | ≈ 11,600/s. |
| Peak per second | 116 K / s | Campaigns create sharp spikes: plan for the peak. |
| SMS cost per day ($) | 160 K | 20 M SMS × $0.008 = $160 K a day: SMS is the expensive channel. |
| Delivery logs per day | 500 GB | Keep detailed logs ~30–90 days, aggregates longer. |
- SMS costs real money per message, so prefer push and email where users allow it.
- Providers have their own rate limits (e.g. per sender number), which shape the worker design.
The estimate that surprises people is cost. A bug that sends every SMS twice costs $160 K a day. That's why de-duplication and pruning dead phone numbers are business requirements here, not polish.
3. API
Write the calls. What must the "send" call contain so a caller can retry safely? Should it return after the message is delivered, or earlier?
Compare with our answer
A strong answer mentions: POST /notifications returning 202 Accepted with an id (delivery is asynchronous) · an idempotency key chosen by the caller · a template id and parameters, not finished text · a priority · a status endpoint · preferences and device registration as separate resources.
| Call | Request | Response | Notes |
|---|---|---|---|
POST /v1/notifications | { userId, template, params, channels?, priority, idempotencyKey } | 202 { notificationId } | Accepted for async delivery; same key = same notification |
POST /v1/campaigns | { segmentId, template, schedule } | 202 { campaignId } | Expanded into per-user notifications by workers |
GET /v1/notifications/{id} | – | { status, channel, attempts[] } | Delivery tracking |
PUT /v1/users/{id}/preferences | { channels, topics, quietHours } | 200 | Opt-outs and quiet hours |
POST /v1/devices | { userId, platform, pushToken } | 201 | Register or refresh a push token |
Two choices worth explaining:
- 202, not 200. The API records the request and queues it; delivery happens later. Making callers wait for an SMS provider would couple every product's latency to Twilio's.
- Template plus parameters, not finished text.
{template: "order_shipped", params: {orderId: 812}}lets the platform pick the user's language, the right channel format (a 160-character SMS vs an HTML email) and the current template version. Callers don't need to know any of that.
4. Data model
Which data is read on every send, which is written on every send, and which database fits each?
Compare with our answer
A strong answer mentions: preferences read on every send, so cached · devices (push tokens) per user, cleaned up when providers reject them · versioned, localised templates · an append-heavy delivery log in a wide-column store with a TTL · idempotency keys in Redis with a TTL.
preferences Relational (PostgreSQL) + cache
user_idchannel_opt_in {push, email, sms}topic_opt_out[]quiet_hours, timezone
Read on every send, written rarely: cache aggressively.
devices Key-value / wide-column (by user_id)
user_id (partition)device_id, platform, push_tokenlast_seen
A user may have several devices; tokens expire and must be cleaned up when providers reject them.
templates Relational + versioning
template_id, version, localechannel, subject, body with {placeholders}
Owned by product teams; localised; must render the same way every time.
notification log Cassandra (by notification_id, and by user/day)
notification_id, user_id, channelstatus, attempts, provider_msg_idcreated_at, sent_at, delivered_at
Billions of rows, append-heavy, queried by id or recent history per user.
dedupe keys Redis with TTL
idem:{key} → notificationId (TTL 48 h)
Fast “have I already accepted this idempotency key?” checks for 24–48 hours.
Comments on the data:
- Preferences are tiny, read on every send and written rarely: a relational table with a cache in front, invalidated by a change event. The cache can be slightly stale for marketing, but an opt-out must take effect quickly (and legally, within a set time).
- Devices are many per user, and push tokens go stale when an app is uninstalled. APNs and FCM tell you when a token is invalid; delete it then, or you'll keep paying to fail.
- The delivery log is billions of rows, written several times per notification (queued, sent, delivered, opened). A wide-column store such as Cassandra, keyed by
notification_idwith a second table by(user_id, day)for "what did we send this user?", with a TTL to expire old rows.
5. High-level design
Sketch the path of a login code and the path of a 10 M-recipient campaign. Where do they share resources, and where must they not?
Compare with our answer
Compare with the figures below. They may share the API and the channel adapters; they must not share a queue or a worker pool. The pieces people forget: the preference check before enqueueing, the delivery log fed by provider callbacks, and campaign expansion as its own batch step.
The architecture, then a login code on the urgent path
- The architecture: product services call one Notify API. It checks preferences and templates, records the request, and puts jobs on separate urgent, normal and bulk queues. Each queue has its own workers, and channel adapters call outside providers like APNs, an email service and an SMS gateway.
- An OTP: the auth service calls the API with an idempotency key. The API checks preferences, renders the template, and puts it on the urgent queue, which its own workers drain within a second or two.
| Component | What it does |
|---|---|
| Notify API | Validates, de-duplicates by idempotency key, checks preferences, renders templates, enqueues |
| Preferences service | Opt-ins, opt-outs, quiet hours, cached |
| Template service | Versioned, localised templates |
| Urgent / normal / bulk queues | Separate queues and worker pools by priority, so a campaign can never delay a login code |
| Channel adapters | One per channel (push, email, SMS): provider rate limits, retries with backoff, circuit breakers and failover |
| Providers | APNs/FCM, email (SES/SendGrid), SMS (Twilio etc.) |
| Delivery log | Every attempt and status, plus provider callbacks |
One notification on two channels, then a campaign that can't delay a login code
- A normal transactional notification can go out on several channels. One job per channel goes on the normal queue, the adapters call each provider, and every result is written to the delivery log.
- A campaign is expanded by workers into ten million jobs on the bulk queue. Because it has its own queue and capacity, a login code sent at the same moment still jumps straight through the urgent queue.
The flow for one send, in order:
- Validate and de-duplicate: has this idempotency key been seen? If so, return the existing id.
- Check preferences: which channels are allowed for this topic, is it quiet hours in the user's time zone, has this user hit their frequency cap?
- Render the template for each allowed channel in the user's language.
- Record the notification as
queuedand enqueue one job per channel on the queue for its priority. - A worker takes the job and hands it to the channel adapter, which applies the provider's rate limit, calls the provider, and records the result.
- Later, provider callbacks (webhooks: "delivered", "bounced", "invalid token") update the log.
6. Deep dives
Write two or three sentences on each before reading: How does an OTP avoid waiting behind a campaign? How do you stop a caller's retry producing two SMS? What do workers do when the SMS provider fails for 5 minutes? What stops ten teams from spamming one user?
Compare with our answer
A strong answer mentions:
- Separate queues and worker pools per priority, plus reserved provider capacity for urgent sends.
- Idempotency keys at the API, de-duplication in workers, and honesty that a crash mid-send can still duplicate.
- Retry with exponential backoff and jitter, a circuit breaker per provider, failover, a dead-letter queue, and expiry for stale OTPs.
- Central preferences, quiet hours and per-user frequency caps, with security messages exempt from opt-out.
Priorities: an OTP must never wait behind a campaign
The problem. A campaign has just put 10 M emails in front of the workers. A user requests a login code. With one first-in-first-out queue, the code arrives after the campaign, perhaps an hour later.
| Option | How | Catch |
|---|---|---|
| One queue | Everything in arrival order | The OTP waits behind 10 M jobs |
| One queue with a priority field | Workers pick the highest priority first | Kafka and SQS standard queues have no priorities; a single broker is still a shared bottleneck |
| Separate queues and worker pools (chosen) | Urgent, normal and bulk queues, each with its own workers and capacity | More moving parts; each pool must be sized |
The choice: isolate by priority, and often by channel too (an SMS outage shouldn't back up push). The urgent pool is small but always has spare capacity; the bulk pool is large and runs at whatever rate the providers allow.
Never twice: idempotency from end to end
The problem. The order service calls POST /notifications, the response times out, and it retries. Without care, the customer gets two "order shipped" emails, or two SMS at $0.008 each.
The defence has three layers:
- At the API: the caller sends an idempotency key (for example
order-812-shipped). The API doesSET idem:{key} {notificationId} NX EX 172800in Redis (set only if absent, expire in 48 hours). If the key exists, it returns the existing notification id instead of creating a new one. - In the queue: queues deliver at least once, so a job can arrive twice (a worker crashes after sending but before acknowledging). The worker checks the log: if this
(notification_id, channel)is alreadysent, it skips. - At the provider: where a provider accepts an idempotency key or a client reference, pass the notification id so the provider can drop a repeat. Many SMS and push APIs don't offer this.
Provider failures: retries, failover and dead letters
The problem. The SMS provider returns errors for 5 minutes. Thousands of OTPs are in flight.
First, classify the error. Retryable: timeouts, HTTP 5xx, "rate limited" (429). Permanent: invalid phone number, unregistered push token, hard email bounce. Retrying a permanent error wastes money; dropping a retryable one loses a message.
| Mechanism | What it does |
|---|---|
| Exponential backoff with jitter | Retry after ~1 s, 2 s, 4 s… plus a random spread, so thousands of workers don't retry in lockstep |
| Circuit breaker per provider | After an error rate threshold, stop calling the provider for a while and fail fast (see circuit breakers and retries) |
| Failover to a second provider | Route SMS to a backup gateway while the circuit is open; for push, there is no backup for APNs, so fall back to another channel if the topic allows |
| Dead-letter queue | After N attempts, park the job with its error for inspection and alerting, instead of retrying forever or dropping it silently |
| Expiry | Every job carries a deadline; an OTP older than its validity (say 5 minutes) is dropped, not delivered late |
Protecting the user: preferences, quiet hours and frequency caps
Ten teams each think their message is important. The platform is the one place that sees all of them, so it enforces:
- Opt-outs per channel and per topic ("no marketing email", "no SMS at all").
- Quiet hours in the user's own time zone: non-urgent messages are delayed until morning, not dropped.
- Frequency caps: at most, say, 3 marketing pushes a day per user, using a per-user counter with a 24-hour window (the same rate limiting idea, keyed by user instead of by caller).
- Exemptions: security messages (login codes, password changes) ignore opt-outs and quiet hours, but still have a sanity limit, so a bug can't send 1,000 codes.
Check preferences twice for scheduled sends: once when the campaign is expanded, and again just before sending, because a user may opt out in between. The law cares about the moment of sending.
Compliance in practice
Marketing email needs a working unsubscribe link (CAN-SPAM in the US, GDPR and ePrivacy rules in the EU), and since 2024 Gmail and Yahoo require one-click unsubscribe from bulk senders. SMS in the US must honour replies such as "STOP", which the provider reports to you by webhook; feed it straight into preferences. Email complaints ("this is spam") and hard bounces go on a suppression list that is checked before every send, because a high bounce or complaint rate damages your sending domain's reputation and gets all your email, including password resets, sent to spam.
Templates and campaigns
Templates are versioned and localised: (template_id, version, locale, channel). Pin the version when a notification is accepted, so a template deployed mid-campaign doesn't produce a mix of old and new messages, and so a bad template can be rolled back. Validate parameters at the API: a missing {orderId} should fail the request, not send "Your order {orderId} has shipped".
Campaigns are a batch job, not a loop in the API. A campaign worker reads the segment (10 M user ids) in pages, filters by preferences, and writes jobs to the bulk queue in batches of a few thousand. For "9 am in each user's time zone", it groups users by time zone and schedules each group separately. The bulk workers then drain at the providers' rate limits: 10 M emails at 5,000 a second takes about 35 minutes, which the requirement allows.
Delivery tracking
"Sent" only means the provider accepted the request. Real status comes later from callbacks: APNs and FCM report invalid tokens, email providers report delivered, bounced and complained, SMS gateways report delivery receipts from carriers. Opens come from the app (push) or a tracking pixel (email, increasingly unreliable since mail clients pre-load images). Write each event to the delivery log and act on the negative ones: delete dead tokens, suppress bouncing addresses.
7. Bottlenecks, failures and monitoring
Scaling bottlenecks
| Bottleneck | Fix |
|---|---|
| Campaign spikes | Expand in batches, bulk queue, autoscale workers, respect provider limits |
| Provider rate limits | Token buckets per provider/sender number; spread across numbers and providers |
| Preference lookups on every send | Cache preferences; push changes via events |
| Log volume (billions of rows) | Wide-column store, TTL old rows, aggregate to daily stats |
When things fail
| Failure | What happens |
|---|---|
| Provider outage | Circuit breaker opens; fail over to backup provider; retry later |
| Worker crashes mid-send | Job is not acked, so it’s redelivered; worker de-dupes before sending |
| Queue backlog grows | Urgent queue protected; bulk degrades gracefully; alert on lag |
| Bad template deployed | Template versions + validation; roll back to the previous version |
What to monitor
Urgent end-to-end latencyQueue depth and lag per priorityProvider error rateDelivery and open ratesDead-letter queue sizeSMS spend per day
8. Follow-up questions
Answer in order before opening ours. 1) The email provider is down for 2 hours during a 10 M campaign. What happens? 2) A bug in one service calls the API in a loop and sends 50,000 requests a minute for one user. What protects the user? 3) Product wants an in-app inbox showing every notification from the last 90 days. What changes?
Compare with our answer
A strong answer mentions:
- The circuit breaker opens, bulk jobs back off and stay queued (or fail over to a backup provider within its limits); urgent email is protected by its own queue; marketing jobs get an expiry so a "flash sale ends tonight" email isn't sent tomorrow; alert on queue age.
- Per-caller rate limits at the API, idempotency keys (if the loop reuses one), and per-user frequency caps that hold even for transactional topics; the sanity limit catches the rest.
- A new channel, "inbox", that writes to a per-user store partitioned by
user_idand sorted by time; unread counts as counters; the app fetches it with cursor pagination; real-time updates via the existing push or a WebSocket.
How would you schedule a notification for 9 am in each user’s timezone?
Store it with a send-at time per user; a scheduler moves due jobs onto the queue, bucketing by minute.
How do you handle a user with 5 devices?
Fan out to all active device tokens; drop tokens the provider says are invalid.
How do you add a new channel, like WhatsApp?
Add a worker type and provider adapter behind the same queue interface: the API and callers don’t change.
- Numbers: 1 B/day ≈ 11.6 K/s avg, ~116 K/s campaign peaks; 2% SMS ≈ $160 K/day; ~500 GB of logs/day.
- API:
POST /notifications→ 202 + id; idempotency key; template + params; priority. - Pipeline: dedupe → preferences, quiet hours, frequency caps → render → log
queued→ queue by priority → worker → channel adapter → provider → callbacks. - Priorities: separate queues and worker pools; reserve provider capacity for urgent; quota who may use urgent.
- Idempotency: Redis
SET NX EXat the API; worker checks the log; duplicates rare, not impossible. - Failures: classify retryable vs permanent; backoff + jitter; circuit breaker; failover; dead-letter queue; expiry for stale OTPs.
- User protection: central opt-outs, quiet hours in local time, frequency caps; security messages exempt but sanity-limited; check again at send time.
- Hygiene: delete invalid push tokens; suppression list for bounces and complaints; SMS STOP handling.
1. A 10 M-email campaign is draining when a user requests a login code by SMS. Both use the same queue and workers. What is the best fix?
Isolation guarantees the urgent path has its own capacity. A priority field doesn't help with queues that have no priorities, and sending without a queue loses retries and buffering.
2. The order service retries POST /notifications after a timeout, and customers get two emails. What stops this at the API?
The key identifies "this same request", not "a similar message". A per-user time window would also block legitimate different notifications.
3. A worker calls the SMS provider, the SMS is sent, then the worker crashes before recording it. The job is redelivered. What is true?
After a crash you can't know whether the provider accepted the request. At-least-once with rare duplicates is the honest guarantee.
4. The SMS provider returns errors for 5 minutes. Which response is best?
Backoff and a circuit breaker stop you hammering a sick provider; failover keeps urgent traffic flowing; dead letters ensure nothing is lost silently.
5. 1 B notifications a day, 2% by SMS at $0.008 each. What is the SMS bill per day, roughly?
20 M SMS × $0.008 = $160 K a day. This is why duplicate SMS and dead phone numbers are real money.
6. A marketing campaign is scheduled for 9 am tomorrow. Tonight, a user in it opts out of marketing email. What should happen?
Preferences must be checked at the moment of sending. Re-expanding a 10 M campaign per opt-out would be wasteful.
7. APNs keeps reporting that 8% of your push tokens are unregistered. What should you do?
An unregistered token means the app was uninstalled or the token rotated; it will never work again. Silently moving users to SMS would cost money and may break their preferences.
Score 60% or more and the lesson is marked complete.