Talking between services · Chapter 11

Message queues

How message queues decouple slow work: delivery guarantees, acks and visibility timeouts, retries and dead-letter queues, ordering, backpressure, and Kafka vs SQS vs RabbitMQ.

IntermediateInterview priority: High30 min
On this page
  1. The problem: checkout that does everything while the customer waits
  2. The idea: write the job down, answer, do it later
  3. How delivery works: receive, process, acknowledge
  4. Delivery semantics
  5. Retries and dead-letter queues
  6. Ordering
  7. Backpressure: what happens when consumers can't keep up
  8. Kafka vs SQS vs RabbitMQ
  9. When not to use a queue
  10. In the interview
  11. Quiz

The problem: checkout that does everything while the customer waits

When a customer clicks "Place order", the checkout service saves the order, charges the card, sends a confirmation email, generates a PDF invoice, updates the warehouse and notifies the analytics system. Done one after another, that takes 3–4 seconds. Worse, if the email provider is down, checkout fails, even though the email could have been sent a minute later.

Do the essential part now and the rest a moment later

  1. When you place an order, the app must save it, charge your card, send a confirmation email, and update the warehouse. If it does all of that while you wait, checkout takes seconds.
  2. Most of that work doesn’t need to happen before you see “Order placed!”. A message queue lets the app write down a job and answer you immediately. The slow work happens a moment later.

Only two steps (save the order, charge the card) must finish before the customer sees "Order placed". Everything else can happen shortly afterwards, as long as it is guaranteed to happen.

The idea: write the job down, answer, do it later

A message queue is a durable buffer between programs. A producer puts a message (a small record describing a job, like {"type": "send_receipt", "order_id": 7731}) on the queue and moves on. Consumers (workers) take messages off and do the work at their own pace.

Producers and consumers only know the queue, which absorbs spikes

  1. The app that adds jobs is the producer. The programs that take jobs off the queue and do them are consumers, or workers. They don’t need to know about each other, only about the queue.
  2. Queues are brilliant at absorbing spikes. During a flash sale, jobs pile up in the queue instead of crashing anything, and workers steadily work through the backlog.

This gives three things at once:

  • Fast responses: the user waits only for the essential work.
  • Decoupling: the producer doesn't know or care which workers exist, how many, or whether they are up right now.
  • Load levelling: a burst of 10× traffic becomes a longer queue, not a crashed email service. Workers drain it at a steady rate.

How delivery works: receive, process, acknowledge

The central question for any queue is: when is a message considered done?

Visibility timeouts: a slow worker means a second delivery

  1. A worker receives a message. The queue doesn't delete it yet; it marks it as in progress. In Amazon SQS it becomes invisible to other workers for the visibility timeout (30 s by default, up to 12 hours). In RabbitMQ it is "unacknowledged" until the worker acks it or its connection closes.
  2. The worker does the job.
  3. The worker acknowledges (acks) it: SQS DeleteMessage, RabbitMQ basic.ack. Only now is the message gone.
  4. If no ack arrives (the worker crashed, or took longer than the timeout), the message is delivered again, possibly to another worker.

This is what makes queues reliable: a crashed worker can't lose a job. It is also why the same job can run twice.

Delivery semantics

GuaranteeHow it happensRiskUse for
At-most-onceAck (or delete) before processingA crash mid-job loses itMetrics samples, non-essential logs
At-least-onceAck after processingA crash after the work but before the ack → done twiceThe default for almost everything
Exactly-onceNot achievable end to end over a network on its own
Effectively-onceAt-least-once delivery + an idempotent consumerNeeds a dedup key or idempotent operationPayments, emails, anything that must not repeat

"Exactly-once" features exist but are narrower than they sound. SQS FIFO queues drop duplicate sends with the same deduplication id within a 5-minute window. Kafka's transactions give exactly-once from one Kafka topic to another. Neither stops your worker from sending an email twice if it crashes after sending and before acknowledging. So the consumer must be idempotent: processing a message twice has the same effect as once. Typical techniques: record processed message ids in a table with a unique constraint, in the same transaction as the work; use operations that are naturally idempotent ("set status = shipped", not "add 1"); pass an idempotency key to the downstream API (see duplicates and idempotency).

Retries and dead-letter queues

Failed jobs are retried; jobs that never succeed go to a dead-letter queue

  1. Jobs sometimes fail, perhaps because the email service is down. The queue simply hands the job out again later: retries. Because a job might run twice, workers should be idempotent: doing it twice has the same effect as once.
  2. Some jobs can never succeed, like an email to an address that doesn’t exist. After a few tries they go to a dead-letter queue, set aside for a human to look at, so they don’t clog everything forever.

Some failures are temporary (the email API returned 503) and some are permanent (the address doesn't exist, or the message is malformed). A message that fails every time is a poison message: retried forever, it wastes workers and can block others behind it.

  • Retry with exponential backoff: wait 1 s, 2 s, 4 s, 8 s… plus random jitter so thousands of failed jobs don't retry in the same instant. In SQS you can change a message's visibility timeout to delay the retry; RabbitMQ setups often use a delay queue.
  • Cap the attempts: after N receives (SQS maxReceiveCount in a redrive policy, often 3–5), move the message to a dead-letter queue (DLQ): a separate queue for messages that couldn't be processed.
  • Watch the DLQ: alert when it's non-empty, inspect, fix the bug, then redrive (replay) the messages to the main queue.

Ordering

Strict order and parallelism pull against each other. If 10 workers take messages from one queue, message 2 can finish before message 1 even if the queue handed them out in order.

QueueOrdering guarantee
SQS standardBest effort only; occasional duplicates and reordering. Nearly unlimited throughput
SQS FIFOStrict order within a message group id; one group is processed one message at a time. Lower throughput (300 messages/s per API action, 3,000 with batching; more in high-throughput mode)
RabbitMQOne queue delivers in order, but with several consumers or redeliveries, processing order is not preserved
KafkaStrict order within a partition; one consumer per partition in a group

The common pattern: you rarely need global order, only order per entity (per order, per user, per account). Use the entity id as the message group id (SQS FIFO) or partition key (Kafka). Events for order 7731 stay in sequence while different orders are processed in parallel.

Backpressure: what happens when consumers can't keep up

Backpressure is any mechanism that pushes back on producers when the system downstream is full, instead of letting work pile up without limit.

Worked example: a flash sale

Normal traffic is 500 orders/s and workers process 2,000 jobs/s. A flash sale brings 5,000 orders/s for 10 minutes.

  • During the sale the queue grows by 5,000 − 2,000 = 3,000 messages/s. After 600 s: 1.8 million messages waiting.
  • After the sale, arrivals drop back to 500/s, so the backlog drains at 2,000 − 500 = 1,500/s: 1,800,000 / 1,500 = 1,200 s, or 20 minutes.
  • The last order of the sale waits about 1.8 M / 2,000 ≈ 15 minutes for its email (time to work through the queue ahead of it).

Is 15 minutes acceptable for a receipt email? Probably. For a payment capture, maybe not. That is the question to ask.

Tools for keeping it under control:

  • Monitor the age of the oldest message, not just the queue length. "Oldest message is 12 minutes old" says directly how late work is.
  • Autoscale workers on queue depth or age, within what the downstream services (database, email API) can take. More workers are useless if they all hammer the same overloaded database.
  • Bound the queue and, when it's full, make producers slow down or reject (HTTP 429/503) rather than accept work you can't do. Set a retention period (SQS keeps messages up to 14 days) so you know the limit.
  • Prioritise: a separate queue for urgent jobs, so password-reset emails don't wait behind a million marketing emails.

Kafka vs SQS vs RabbitMQ

Amazon SQSRabbitMQKafka
ModelManaged queue; messages deleted when ackedBroker with exchanges routing to queues; deleted when ackedAppend-only log, split into partitions; messages kept for a retention period, consumers track an offset
Replay old messagesNoNoYes, rewind the offset
OrderingFIFO queues: per groupPer queue (single consumer)Per partition
ThroughputVery high (standard); FIFO limitedTens of thousands of msgs/s per node, typicallyVery high: tens of MB/s per partition, millions of msgs/s per cluster
RoutingSimple; fan-out via SNSRich: topic, direct, fan-out exchanges, prioritiesBy topic and partition key
OperationsNone (managed)You run it (or use a managed service)Heavier to run; managed options exist
Pick forBackground jobs on AWS with zero opsTask queues with flexible routing, priorities, per-message acksHigh-volume event streams, several independent readers, replay

A task queue (one job, done once by one worker) suits SQS or RabbitMQ. An event stream read by many services, possibly replayed, suits Kafka; see pub/sub and events.

Why Kafka is not a classic queue

In Kafka, consuming doesn't remove anything. Each consumer group stores how far it has read (its committed offset) per partition. A failed message can't be individually retried or skipped while later ones continue: the consumer either retries in place (blocking that partition) or writes the failed message to a separate retry topic and moves on. That is why "per-message ack with redelivery" workloads (variable-length jobs, priorities) often fit SQS or RabbitMQ better, while Kafka shines for ordered, high-volume streams that several systems read.

When not to use a queue

  • The user needs the result now: a search query or a balance check. A queue adds latency and complexity for nothing.
  • Low volume and simple flow: a direct call with a retry is easier to debug.
  • You need a reply: request–reply over a queue is possible (reply queues, correlation ids) but awkward; prefer a synchronous call.
  • It hides an overload rather than fixing it: if consumers are permanently slower than producers, the queue only delays the failure.

In the interview

Your turn: a video upload pipeline

Users upload videos; each must be transcoded into 4 resolutions (each takes 1–6 minutes) and then the uploader gets a notification. At peak, 200 uploads per minute. Design the queueing: what messages, what timeouts, what happens on failure, and how many workers?

Compare with our answer
  • On upload, store the file in object storage, write a videos row (status processing) and enqueue one message per resolution (800 messages/min at peak) so they run in parallel.
  • Visibility timeout above the worst case (say 10 minutes), or a heartbeat that extends it while transcoding.
  • Idempotent workers: output paths are deterministic (video/123/720p.mp4), so a rerun overwrites the same file; mark each resolution done with a conditional update.
  • When all four are done (count completed resolutions in the database), enqueue a notify message.
  • Retries with backoff; after 3 failures to a DLQ with an alert; mark the video as failed if the original file is corrupt.
  • Workers: 800 jobs/min × ~3.5 min average ≈ 2,800 jobs in progress (Little's law), so around 2,800 concurrent transcodes at peak, autoscaled on queue age.

A strong answer mentions: one message per independent unit of work · visibility timeout above processing time or extended by heartbeat · idempotent outputs · a DLQ and alert · a fan-in step for the final notification · worker count from rate × duration.

Your turn: spot the bug

A worker does this for each message: receive, delete the message, then charge the customer's card and send the receipt. Charges are occasionally missing. Another team's worker acks after processing, and customers occasionally get two receipts. What's wrong in each, and what is the right design?

Compare with our answer
  • Team 1 uses at-most-once: if the worker crashes after deleting but before charging, the job is lost. Delete only after the work succeeds.
  • Team 2 is correctly at-least-once, so duplicates are expected: a crash or timeout after sending the receipt but before acking redelivers the message.
  • The right design: ack after processing, and make the work idempotent: charge with an idempotency key derived from the order id; record "receipt sent for order X" with a unique constraint before or atomically with sending, or use the email provider's idempotency key.
  • Check the visibility timeout covers the processing time, and send repeatedly failing messages to a DLQ.

A strong answer mentions: ack-before-work loses messages · ack-after-work duplicates them · idempotent processing as the fix · idempotency keys to payment and email APIs · visibility timeout and DLQ settings.

Message queues on one page
  • Queue = durable buffer: producer enqueues and answers fast; consumers work at their own pace. Gives speed, decoupling, load levelling.
  • Receive → process → ack. No ack before the visibility timeout (SQS default 30 s) → redelivered.
  • At-most-once (ack first, may lose) · at-least-once (ack after, may duplicate: the default) · effectively-once = at-least-once + idempotent consumer.
  • Retries with exponential backoff + jitter; after N attempts → DLQ; alert on it, fix, redrive.
  • Timeout shorter than the job → duplicate work. Set it above the worst case or extend with a heartbeat.
  • Order per entity: SQS FIFO message group id, Kafka partition key. Parallel consumers break global order.
  • Backpressure: alert on age of oldest message, autoscale within downstream limits, bound the queue, separate urgent work.
  • SQS = zero-ops job queue · RabbitMQ = flexible routing and acks · Kafka = replayable partitioned log for event streams.
Interactive labPractise it: the queue playgroundOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. A worker receives a message, sends a welcome email, and crashes before acknowledging it. What happens with a typical at-least-once queue?

2. Image-resize jobs take up to 3 minutes, but the SQS visibility timeout is 30 seconds. What will you see?

3. A team wants "exactly-once" sending of receipt emails. What is the most realistic design?

4. A queue receives 6,000 jobs/s for 5 minutes, while workers handle 2,000 jobs/s. Afterwards arrivals drop to 1,000 jobs/s. How long does the backlog take to clear?

5. Events for each bank account must be applied in order, but the system handles millions of accounts. What is the best approach?

6. A message with a malformed payload fails every time and keeps being retried, slowing the workers. What should catch it?

7. Several analytics, billing and audit services all need to read the same stream of order events, and audit may need to replay last week's events. Which tool fits best?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.