Message queues
How message queues decouple slow work: delivery guarantees, acks and visibility timeouts, retries and dead-letter queues, ordering, backpressure, and Kafka vs SQS vs RabbitMQ.
On this page
- The problem: checkout that does everything while the customer waits
- The idea: write the job down, answer, do it later
- How delivery works: receive, process, acknowledge
- Delivery semantics
- Retries and dead-letter queues
- Ordering
- Backpressure: what happens when consumers can't keep up
- Kafka vs SQS vs RabbitMQ
- When not to use a queue
- In the interview
- Quiz
The problem: checkout that does everything while the customer waits
When a customer clicks "Place order", the checkout service saves the order, charges the card, sends a confirmation email, generates a PDF invoice, updates the warehouse and notifies the analytics system. Done one after another, that takes 3–4 seconds. Worse, if the email provider is down, checkout fails, even though the email could have been sent a minute later.
Do the essential part now and the rest a moment later
- When you place an order, the app must save it, charge your card, send a confirmation email, and update the warehouse. If it does all of that while you wait, checkout takes seconds.
- Most of that work doesn’t need to happen before you see “Order placed!”. A message queue lets the app write down a job and answer you immediately. The slow work happens a moment later.
Only two steps (save the order, charge the card) must finish before the customer sees "Order placed". Everything else can happen shortly afterwards, as long as it is guaranteed to happen.
The idea: write the job down, answer, do it later
A message queue is a durable buffer between programs. A producer puts a message (a small record describing a job, like {"type": "send_receipt", "order_id": 7731}) on the queue and moves on. Consumers (workers) take messages off and do the work at their own pace.
Producers and consumers only know the queue, which absorbs spikes
- The app that adds jobs is the producer. The programs that take jobs off the queue and do them are consumers, or workers. They don’t need to know about each other, only about the queue.
- Queues are brilliant at absorbing spikes. During a flash sale, jobs pile up in the queue instead of crashing anything, and workers steadily work through the backlog.
This gives three things at once:
- Fast responses: the user waits only for the essential work.
- Decoupling: the producer doesn't know or care which workers exist, how many, or whether they are up right now.
- Load levelling: a burst of 10× traffic becomes a longer queue, not a crashed email service. Workers drain it at a steady rate.
How delivery works: receive, process, acknowledge
The central question for any queue is: when is a message considered done?
Visibility timeouts: a slow worker means a second delivery
- A worker receives a message. The queue doesn't delete it yet; it marks it as in progress. In Amazon SQS it becomes invisible to other workers for the visibility timeout (30 s by default, up to 12 hours). In RabbitMQ it is "unacknowledged" until the worker acks it or its connection closes.
- The worker does the job.
- The worker acknowledges (acks) it: SQS
DeleteMessage, RabbitMQbasic.ack. Only now is the message gone. - If no ack arrives (the worker crashed, or took longer than the timeout), the message is delivered again, possibly to another worker.
This is what makes queues reliable: a crashed worker can't lose a job. It is also why the same job can run twice.
Delivery semantics
| Guarantee | How it happens | Risk | Use for |
|---|---|---|---|
| At-most-once | Ack (or delete) before processing | A crash mid-job loses it | Metrics samples, non-essential logs |
| At-least-once | Ack after processing | A crash after the work but before the ack → done twice | The default for almost everything |
| Exactly-once | Not achievable end to end over a network on its own | ||
| Effectively-once | At-least-once delivery + an idempotent consumer | Needs a dedup key or idempotent operation | Payments, emails, anything that must not repeat |
"Exactly-once" features exist but are narrower than they sound. SQS FIFO queues drop duplicate sends with the same deduplication id within a 5-minute window. Kafka's transactions give exactly-once from one Kafka topic to another. Neither stops your worker from sending an email twice if it crashes after sending and before acknowledging. So the consumer must be idempotent: processing a message twice has the same effect as once. Typical techniques: record processed message ids in a table with a unique constraint, in the same transaction as the work; use operations that are naturally idempotent ("set status = shipped", not "add 1"); pass an idempotency key to the downstream API (see duplicates and idempotency).
Retries and dead-letter queues
Failed jobs are retried; jobs that never succeed go to a dead-letter queue
- Jobs sometimes fail, perhaps because the email service is down. The queue simply hands the job out again later: retries. Because a job might run twice, workers should be idempotent: doing it twice has the same effect as once.
- Some jobs can never succeed, like an email to an address that doesn’t exist. After a few tries they go to a dead-letter queue, set aside for a human to look at, so they don’t clog everything forever.
Some failures are temporary (the email API returned 503) and some are permanent (the address doesn't exist, or the message is malformed). A message that fails every time is a poison message: retried forever, it wastes workers and can block others behind it.
- Retry with exponential backoff: wait 1 s, 2 s, 4 s, 8 s… plus random jitter so thousands of failed jobs don't retry in the same instant. In SQS you can change a message's visibility timeout to delay the retry; RabbitMQ setups often use a delay queue.
- Cap the attempts: after N receives (SQS
maxReceiveCountin a redrive policy, often 3–5), move the message to a dead-letter queue (DLQ): a separate queue for messages that couldn't be processed. - Watch the DLQ: alert when it's non-empty, inspect, fix the bug, then redrive (replay) the messages to the main queue.
Ordering
Strict order and parallelism pull against each other. If 10 workers take messages from one queue, message 2 can finish before message 1 even if the queue handed them out in order.
| Queue | Ordering guarantee |
|---|---|
| SQS standard | Best effort only; occasional duplicates and reordering. Nearly unlimited throughput |
| SQS FIFO | Strict order within a message group id; one group is processed one message at a time. Lower throughput (300 messages/s per API action, 3,000 with batching; more in high-throughput mode) |
| RabbitMQ | One queue delivers in order, but with several consumers or redeliveries, processing order is not preserved |
| Kafka | Strict order within a partition; one consumer per partition in a group |
The common pattern: you rarely need global order, only order per entity (per order, per user, per account). Use the entity id as the message group id (SQS FIFO) or partition key (Kafka). Events for order 7731 stay in sequence while different orders are processed in parallel.
Backpressure: what happens when consumers can't keep up
Backpressure is any mechanism that pushes back on producers when the system downstream is full, instead of letting work pile up without limit.
Worked example: a flash sale
Normal traffic is 500 orders/s and workers process 2,000 jobs/s. A flash sale brings 5,000 orders/s for 10 minutes.
- During the sale the queue grows by 5,000 − 2,000 = 3,000 messages/s. After 600 s: 1.8 million messages waiting.
- After the sale, arrivals drop back to 500/s, so the backlog drains at 2,000 − 500 = 1,500/s: 1,800,000 / 1,500 = 1,200 s, or 20 minutes.
- The last order of the sale waits about 1.8 M / 2,000 ≈ 15 minutes for its email (time to work through the queue ahead of it).
Is 15 minutes acceptable for a receipt email? Probably. For a payment capture, maybe not. That is the question to ask.
Tools for keeping it under control:
- Monitor the age of the oldest message, not just the queue length. "Oldest message is 12 minutes old" says directly how late work is.
- Autoscale workers on queue depth or age, within what the downstream services (database, email API) can take. More workers are useless if they all hammer the same overloaded database.
- Bound the queue and, when it's full, make producers slow down or reject (HTTP 429/503) rather than accept work you can't do. Set a retention period (SQS keeps messages up to 14 days) so you know the limit.
- Prioritise: a separate queue for urgent jobs, so password-reset emails don't wait behind a million marketing emails.
Kafka vs SQS vs RabbitMQ
| Amazon SQS | RabbitMQ | Kafka | |
|---|---|---|---|
| Model | Managed queue; messages deleted when acked | Broker with exchanges routing to queues; deleted when acked | Append-only log, split into partitions; messages kept for a retention period, consumers track an offset |
| Replay old messages | No | No | Yes, rewind the offset |
| Ordering | FIFO queues: per group | Per queue (single consumer) | Per partition |
| Throughput | Very high (standard); FIFO limited | Tens of thousands of msgs/s per node, typically | Very high: tens of MB/s per partition, millions of msgs/s per cluster |
| Routing | Simple; fan-out via SNS | Rich: topic, direct, fan-out exchanges, priorities | By topic and partition key |
| Operations | None (managed) | You run it (or use a managed service) | Heavier to run; managed options exist |
| Pick for | Background jobs on AWS with zero ops | Task queues with flexible routing, priorities, per-message acks | High-volume event streams, several independent readers, replay |
A task queue (one job, done once by one worker) suits SQS or RabbitMQ. An event stream read by many services, possibly replayed, suits Kafka; see pub/sub and events.
Why Kafka is not a classic queue
In Kafka, consuming doesn't remove anything. Each consumer group stores how far it has read (its committed offset) per partition. A failed message can't be individually retried or skipped while later ones continue: the consumer either retries in place (blocking that partition) or writes the failed message to a separate retry topic and moves on. That is why "per-message ack with redelivery" workloads (variable-length jobs, priorities) often fit SQS or RabbitMQ better, while Kafka shines for ordered, high-volume streams that several systems read.
When not to use a queue
- The user needs the result now: a search query or a balance check. A queue adds latency and complexity for nothing.
- Low volume and simple flow: a direct call with a retry is easier to debug.
- You need a reply: request–reply over a queue is possible (reply queues, correlation ids) but awkward; prefer a synchronous call.
- It hides an overload rather than fixing it: if consumers are permanently slower than producers, the queue only delays the failure.
In the interview
Users upload videos; each must be transcoded into 4 resolutions (each takes 1–6 minutes) and then the uploader gets a notification. At peak, 200 uploads per minute. Design the queueing: what messages, what timeouts, what happens on failure, and how many workers?
Compare with our answer
- On upload, store the file in object storage, write a
videosrow (statusprocessing) and enqueue one message per resolution (800 messages/min at peak) so they run in parallel. - Visibility timeout above the worst case (say 10 minutes), or a heartbeat that extends it while transcoding.
- Idempotent workers: output paths are deterministic (
video/123/720p.mp4), so a rerun overwrites the same file; mark each resolution done with a conditional update. - When all four are done (count completed resolutions in the database), enqueue a
notifymessage. - Retries with backoff; after 3 failures to a DLQ with an alert; mark the video as failed if the original file is corrupt.
- Workers: 800 jobs/min × ~3.5 min average ≈ 2,800 jobs in progress (Little's law), so around 2,800 concurrent transcodes at peak, autoscaled on queue age.
A strong answer mentions: one message per independent unit of work · visibility timeout above processing time or extended by heartbeat · idempotent outputs · a DLQ and alert · a fan-in step for the final notification · worker count from rate × duration.
A worker does this for each message: receive, delete the message, then charge the customer's card and send the receipt. Charges are occasionally missing. Another team's worker acks after processing, and customers occasionally get two receipts. What's wrong in each, and what is the right design?
Compare with our answer
- Team 1 uses at-most-once: if the worker crashes after deleting but before charging, the job is lost. Delete only after the work succeeds.
- Team 2 is correctly at-least-once, so duplicates are expected: a crash or timeout after sending the receipt but before acking redelivers the message.
- The right design: ack after processing, and make the work idempotent: charge with an idempotency key derived from the order id; record "receipt sent for order X" with a unique constraint before or atomically with sending, or use the email provider's idempotency key.
- Check the visibility timeout covers the processing time, and send repeatedly failing messages to a DLQ.
A strong answer mentions: ack-before-work loses messages · ack-after-work duplicates them · idempotent processing as the fix · idempotency keys to payment and email APIs · visibility timeout and DLQ settings.
- Queue = durable buffer: producer enqueues and answers fast; consumers work at their own pace. Gives speed, decoupling, load levelling.
- Receive → process → ack. No ack before the visibility timeout (SQS default 30 s) → redelivered.
- At-most-once (ack first, may lose) · at-least-once (ack after, may duplicate: the default) · effectively-once = at-least-once + idempotent consumer.
- Retries with exponential backoff + jitter; after N attempts → DLQ; alert on it, fix, redrive.
- Timeout shorter than the job → duplicate work. Set it above the worst case or extend with a heartbeat.
- Order per entity: SQS FIFO message group id, Kafka partition key. Parallel consumers break global order.
- Backpressure: alert on age of oldest message, autoscale within downstream limits, bound the queue, separate urgent work.
- SQS = zero-ops job queue · RabbitMQ = flexible routing and acks · Kafka = replayable partitioned log for event streams.
1. A worker receives a message, sends a welcome email, and crashes before acknowledging it. What happens with a typical at-least-once queue?
No ack means redelivery after the visibility timeout. The work already happened, so the email goes out again unless the consumer is idempotent.
2. Image-resize jobs take up to 3 minutes, but the SQS visibility timeout is 30 seconds. What will you see?
When the timeout expires, the message becomes visible and another worker takes it. Set the timeout above the worst case or extend it while working.
3. A team wants "exactly-once" sending of receipt emails. What is the most realistic design?
Delivery can repeat whatever the queue promises, so the consumer must dedupe. FIFO deduplication covers duplicate sends to the queue, and Kafka transactions don't include an external email API.
4. A queue receives 6,000 jobs/s for 5 minutes, while workers handle 2,000 jobs/s. Afterwards arrivals drop to 1,000 jobs/s. How long does the backlog take to clear?
Backlog = (6,000 − 2,000) × 300 s = 1.2 M. Spare capacity afterwards is 2,000 − 1,000 = 1,000/s, so 1,200 s = 20 minutes.
5. Events for each bank account must be applied in order, but the system handles millions of accounts. What is the best approach?
Only per-account order matters. A partition key or FIFO group id gives that with parallelism across accounts. A single consumer can't keep up; millions of queues are unmanageable.
6. A message with a malformed payload fails every time and keeps being retried, slowing the workers. What should catch it?
A poison message never succeeds. Capping attempts and moving it to a DLQ protects the workers; the alert gets a human to fix the cause.
7. Several analytics, billing and audit services all need to read the same stream of order events, and audit may need to replay last week's events. Which tool fits best?
Kafka keeps events for a retention period and lets each consumer group track its own offset, so many readers and replay come naturally. A shared queue would give each event to only one of the services.
Score 60% or more and the lesson is marked complete.