Long-running and background jobs
How to move slow work out of the request: job queues and workers, status by polling or webhooks, retries with backoff, dead-letter queues, leases and scheduled jobs.
On this page
The problem: a request that takes four minutes
A video site lets users upload clips. After the upload, the server must transcode the file into five resolutions, extract thumbnails and run a content check. On a good day that takes 2–4 minutes of CPU.
The first version does all of it inside the POST /videos request. It fails in three ways at once:
- Timeouts: load balancers and browsers give up on a quiet connection after 30–120 seconds. The user sees an error even though the work is still running.
- Duplicate work: the user (or the client library) retries, so the server starts transcoding the same file a second time.
- Lost work and wasted capacity: a deploy or crash kills the process mid-transcode and nothing remembers to restart it. Meanwhile each slow request ties up a web server thread that could have served hundreds of fast pages.
The fix is to separate accepting the work from doing it.
Signals that you are facing it
- An operation takes more than a second or two: media processing, report generation, bulk imports, sending 100,000 emails, calling a slow third party.
- The user does not need the result immediately: "we'll email you when your export is ready".
- Work must survive crashes and be retried when a dependency fails.
- Something must happen later or on a schedule: "remind the driver in 10 minutes", "bill every customer on the 1st".
- Load arrives in bursts that you would rather smooth out than provision for.
Say early: "This doesn't belong in the request path. I'll accept it, store a job, return a job ID, and let workers do it asynchronously."
The idea: accept fast, work later, report back
Moving slow work out of the request into a job queue
The request path does only cheap, safe things: validate the input, store the uploaded file, write a job (a record describing the work: type, arguments, attempt count, status) and reply 202 Accepted with a job ID. A worker is a separate process that takes jobs from a queue (a durable list of pending work, see message queues) and runs them. Web servers and workers scale independently: a burst of uploads lengthens the queue instead of crashing the site.
The client learns the outcome in one of three ways:
| Method | How | Good for | Catch |
|---|---|---|---|
| Polling | GET /jobs/123 every few seconds; server may send Retry-After | Simple clients, any network | Wasted requests; result delayed up to one interval |
| Push to the user | WebSocket or server-sent events when status changes | Live progress bars in an app | Needs a connection layer, see real-time updates |
| Webhook | The system calls a URL the client registered (POST /their-callback) | Server-to-server, e.g. payment providers | Receiver may be down: you must retry, sign the request (HMAC), and the receiver must dedupe |
A typical status response:
{ "id": "job_123", "status": "running", "progress": 0.6,
"attempts": 1, "result_url": null, "error": null }
How it works
Leases and visibility timeouts
A queue must not lose a job if the worker that took it dies. So taking a job does not delete it; it grants a lease (temporary ownership). In Amazon SQS this is the visibility timeout: a received message is hidden from other consumers for, say, 30 seconds (the default; up to 12 hours). If the worker finishes, it deletes the message. If it crashes, the timeout expires and the message becomes visible to another worker.
A lease expires and another worker takes over
Two consequences matter:
- Delivery is at least once. A worker that finishes the work but crashes before deleting the message will see the job run twice. Every job must therefore be idempotent (safe to run more than once), for example by writing results under a deterministic key or checking a "done" marker first. See duplicates and idempotency.
- Long jobs must extend their lease. A job that legitimately takes 10 minutes under a 30-second timeout will be handed to a second worker while the first is still busy. The worker sends a heartbeat (SQS:
ChangeMessageVisibility) every so often to extend the lease.
You don't need a separate queue product to start. Postgres can act as a job queue: FOR UPDATE SKIP LOCKED lets many workers each claim a different row without blocking each other.
UPDATE jobs
SET status = 'running', attempts = attempts + 1,
lease_until = now() + interval '5 minutes'
WHERE id = (
SELECT id FROM jobs
WHERE run_at <= now()
AND (status = 'queued' OR (status = 'running' AND lease_until < now()))
ORDER BY run_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING id, kind, payload, attempts;
The second condition picks up jobs whose worker died. The run_at column also gives you delayed jobs for free.
Retries with backoff, then a dead-letter queue
Failures fall into two kinds. Transient ones (a timeout, an HTTP 503, a deadlock) usually succeed on retry. Permanent ones (a corrupt file, a missing record, a bug) never will. Retry the first kind with exponential backoff (wait 1 s, 2 s, 4 s, 8 s...) plus jitter (randomness, so 10,000 failed jobs don't all retry in the same second). Cap the delay and the number of attempts.
Backoff, then a dead-letter queue for jobs that keep failing
A job that exhausts its attempts goes to a dead-letter queue (DLQ): a holding area for failed jobs, with an alert on its size. A human fixes the cause and replays the jobs back into the main queue. In SQS this is a redrive policy with maxReceiveCount.
import random
BASE_SECONDS, CAP_SECONDS, MAX_ATTEMPTS = 2, 600, 6
def retry_delay(attempt: int) -> float:
# "full jitter": random point between 0 and the exponential ceiling
return random.uniform(0, min(CAP_SECONDS, BASE_SECONDS * 2 ** attempt))
def on_failure(db, job, error, permanent: bool) -> None:
if permanent or job.attempts >= MAX_ATTEMPTS:
db.execute("UPDATE jobs SET status = 'dead', error = %s WHERE id = %s",
(str(error), job.id))
return
db.execute("""UPDATE jobs SET status = 'queued', lease_until = NULL,
run_at = now() + make_interval(secs => %s), error = %s
WHERE id = %s""", (retry_delay(job.attempts), str(error), job.id))
Delayed and recurring jobs
A delayed job runs at a given time: store run_at and only claim jobs whose time has come (above), or keep a Redis sorted set scored by run time and move due entries to the work queue. SQS supports delays of up to 15 minutes per message; longer delays need your own store.
A recurring job ("every night at 02:00") needs a scheduler that enqueues a job each time; the work itself still runs on ordinary workers. Run more than one scheduler for availability and two of them will fire the same slot, so make the enqueue idempotent: a unique key such as nightly-report:2026-11-19 means the second insert fails harmlessly. The distributed job scheduler case study goes deep on this.
Breaking up very long jobs
A 6-hour job that fails at hour five and restarts from zero will rarely finish. Split it: a parent job enqueues one child per chunk (per 1,000 rows, per video segment), and a counter or a final "join" job notices when all children are done. Or checkpoint: save progress every few minutes so a retry resumes rather than restarts.
Worked example: sizing the workers
The site receives 20 uploads a second at peak, and each needs on average 90 CPU-seconds of transcoding.
- Work arriving: 20 × 90 = 1,800 CPU-seconds per second. By Little's law (work in progress = arrival rate × time per job), that is 1,800 jobs running at once.
- At one job per core and 32 cores per machine: 1,800 ÷ 32 ≈ 57 machines, around 70 with 20% headroom.
- Suppose peak lasts 2 hours and you deliberately run only 45 machines (1,440 cores). The shortfall is 360 CPU-seconds per second, about 4 jobs a second, so ~29,000 jobs back up by the end of the peak.
- Off-peak, uploads drop to 10 a second (900 cores busy), leaving 540 spare cores. The backlog of 29,000 × 90 CPU-seconds clears in about 29,000 × 90 ÷ 540 ≈ 4,800 s, roughly 80 minutes.
That trade (cheaper fleet, longer waits at peak) is a product decision. Make it visible: autoscale on queue depth and age of the oldest job, not on CPU, because idle-looking workers can sit beside a growing backlog if they are blocked on I/O.
Options and trade-offs
| Tool | Good at | Weak at | Use when |
|---|---|---|---|
Postgres table + SKIP LOCKED | No new infrastructure; transactional enqueue with your data | Thousands of jobs/s at most; polling load | Small to medium systems |
| Redis-based queue (Sidekiq, BullMQ, Celery) | Fast, simple, delayed jobs built in | Durability depends on Redis persistence settings | Typical web app background work |
| Managed queue (SQS, Cloud Tasks) | No servers; leases, DLQ, scaling built in | 15-minute max delay in SQS; vendor limits | Most cloud systems |
| Kafka as a job queue | Huge throughput, replay | One slow message blocks its partition; no per-message retry or lease | High-volume event processing, not individual slow jobs |
| Workflow engine (Temporal, Step Functions) | Multi-step jobs with durable state, timers, retries per step | Another system to learn and run | Long, multi-step business processes |
One subtle point with a separate queue: if you write the job to the queue and the related row to your database as two steps, a crash between them leaves one without the other. The transactional outbox (write the job into an outbox table in the same transaction, then relay it to the queue) closes that gap; see transactions across services.
When not to use it
- The user needs the answer to continue. A login or a price check should be made fast, not made asynchronous.
- The work takes 200 ms. A queue adds latency, a status endpoint and failure modes. Make the request faster instead.
- Ordering across jobs matters strictly. Workers run in parallel and retries reorder things; if order matters, partition by key and process one key at a time.
Which case studies use it
- Design YouTube: transcoding pipeline, one job per segment and resolution.
- Design a distributed job scheduler: delayed and recurring jobs, leases, exactly-once-looking execution.
- Design a notification system: per-channel queues, retries with backoff, DLQs for bad tokens.
- Design a web crawler: a frontier of URLs as jobs with politeness delays.
- Design a payment system: asynchronous provider callbacks (webhooks) and reconciliation jobs.
- Design Dropbox: post-upload processing such as previews and indexing.
In the interview
Users of an analytics product can export up to 50 million rows as a CSV. Exports take from 10 seconds to 40 minutes. Design the flow from the button click to the user downloading the file, including what happens if a worker dies at minute 30.
Compare with our answer
POST /exportswrites a job row and returns 202 + export ID; the UI pollsGET /exports/{id}(or the user gets an email with a link).- Workers stream rows in chunks (e.g. by primary-key range) to object storage using a multipart upload, saving the last finished chunk as a checkpoint.
- A lease with heartbeats, so a dead worker's job is picked up after the lease expires and resumes from the checkpoint, not row zero.
- The finished file is served with a pre-signed URL that expires; old exports are deleted by a lifecycle rule.
- Limits: one running export per user, a separate queue so huge exports don't delay small ones.
A strong answer mentions: 202 + job ID with polling or email · chunking and checkpoints · lease expiry and heartbeats · idempotent chunk writes · a separate queue or priority for big jobs · pre-signed download URL.
A team sends welcome emails from SQS workers with a 30-second visibility timeout. The email provider sometimes takes 45 seconds to respond. Customers complain about receiving two or three identical welcome emails. Why, and what would you change?
Compare with our answer
The provider call outlives the 30-second lease. The message becomes visible again while the first worker is still waiting, a second worker takes it and sends again; sometimes a third does too. The first worker then deletes the message, but the duplicates have already gone out.
A strong answer mentions:
- The visibility timeout is shorter than the job, so at-least-once delivery turns into routine duplicates.
- Raise the timeout well above the worst case, or extend it with a heartbeat while waiting.
- Make sending idempotent: record "welcome sent for user X" (or pass an idempotency key the provider honours) and check it before sending.
- Set a client timeout on the provider call shorter than the lease.
- Add metrics on receive count > 1 to catch it early.
- Trigger: work over ~1–2 s, bursty load, retries needed, or something scheduled for later.
- Shape: validate → store input → enqueue job →
202 Accepted+ job ID → workers → status via polling, push or webhook. - Queues are at-least-once: jobs must be idempotent.
- Lease / visibility timeout (SQS default 30 s, max 12 h); heartbeat to extend; fencing token against a stale worker.
- Retry transient errors with exponential backoff + jitter, cap attempts (~5–8), then a DLQ with an alert and replay.
- Postgres
FOR UPDATE SKIP LOCKEDis a fine queue at modest scale;run_atgives delays. - Recurring jobs: scheduler enqueues with a unique key per slot; workers do the work.
- Capacity: jobs in flight = rate × duration; autoscale on queue age, separate queues per priority.
1. An SQS worker resizes images in about 5 seconds, but sometimes the same image is processed twice and two thumbnails appear. What is the most likely cause?
Standard queues deliver at least once: a job is only removed when the worker deletes it, so a crash after the work but before the delete repeats it. The fix is idempotent output, such as a deterministic thumbnail key.
2. A report job takes up to 20 minutes, but the queue's visibility timeout is 5 minutes. What should change?
A lease shorter than the job hands it to a second worker mid-run. Extending the lease while working (and idempotent output for the rare overlap) fixes it; more workers do not make one job faster.
3. A payment provider has an outage for 10 minutes. 40,000 jobs that call it fail and are retried every 5 seconds. When the provider recovers, it falls over again. What would have prevented this?
Fixed, synchronised retries form a retry storm. Backoff spreads them out, jitter desynchronises them, and a breaker stops sending while the dependency is known to be down.
4. Three scheduler instances run for high availability, and the nightly billing job is enqueued three times. What is the cleanest fix?
Idempotent enqueueing keeps the availability of three schedulers while making duplicates harmless. A single scheduler is a single point of failure, and offsets still run billing three times.
5. A shop runs 300 bulk-import jobs during the day, each taking 30 minutes. Password-reset emails, processed by the same workers, now take 25 minutes to arrive. What is the best fix?
The emails are stuck behind long jobs: head-of-line blocking. Separate queues (or priorities) isolate fast, urgent work from slow, bulky work.
6. Uploads arrive at 50 per second and each takes 12 seconds of single-core work. Roughly how many cores must be busy at peak to keep up?
Little's law: work in progress = arrival rate × time per job = 50 × 12 = 600 jobs in flight, so about 600 cores, plus headroom.
Score 60% or more and the lesson is marked complete.