Interview patterns · Design 10

Long-running and background jobs

How to move slow work out of the request: job queues and workers, status by polling or webhooks, retries with backoff, dead-letter queues, leases and scheduled jobs.

IntermediateInterview priority: High35 min
On this page
  1. The problem: a request that takes four minutes
  2. Signals that you are facing it
  3. The idea: accept fast, work later, report back
  4. How it works
  5. Worked example: sizing the workers
  6. Options and trade-offs
  7. When not to use it
  8. Which case studies use it
  9. In the interview
  10. Quiz

The problem: a request that takes four minutes

A video site lets users upload clips. After the upload, the server must transcode the file into five resolutions, extract thumbnails and run a content check. On a good day that takes 2–4 minutes of CPU.

The first version does all of it inside the POST /videos request. It fails in three ways at once:

  • Timeouts: load balancers and browsers give up on a quiet connection after 30–120 seconds. The user sees an error even though the work is still running.
  • Duplicate work: the user (or the client library) retries, so the server starts transcoding the same file a second time.
  • Lost work and wasted capacity: a deploy or crash kills the process mid-transcode and nothing remembers to restart it. Meanwhile each slow request ties up a web server thread that could have served hundreds of fast pages.

The fix is to separate accepting the work from doing it.

Signals that you are facing it

  • An operation takes more than a second or two: media processing, report generation, bulk imports, sending 100,000 emails, calling a slow third party.
  • The user does not need the result immediately: "we'll email you when your export is ready".
  • Work must survive crashes and be retried when a dependency fails.
  • Something must happen later or on a schedule: "remind the driver in 10 minutes", "bill every customer on the 1st".
  • Load arrives in bursts that you would rather smooth out than provision for.

Say early: "This doesn't belong in the request path. I'll accept it, store a job, return a job ID, and let workers do it asynchronously."

The idea: accept fast, work later, report back

Moving slow work out of the request into a job queue

The request path does only cheap, safe things: validate the input, store the uploaded file, write a job (a record describing the work: type, arguments, attempt count, status) and reply 202 Accepted with a job ID. A worker is a separate process that takes jobs from a queue (a durable list of pending work, see message queues) and runs them. Web servers and workers scale independently: a burst of uploads lengthens the queue instead of crashing the site.

The client learns the outcome in one of three ways:

MethodHowGood forCatch
PollingGET /jobs/123 every few seconds; server may send Retry-AfterSimple clients, any networkWasted requests; result delayed up to one interval
Push to the userWebSocket or server-sent events when status changesLive progress bars in an appNeeds a connection layer, see real-time updates
WebhookThe system calls a URL the client registered (POST /their-callback)Server-to-server, e.g. payment providersReceiver may be down: you must retry, sign the request (HMAC), and the receiver must dedupe

A typical status response:

{ "id": "job_123", "status": "running", "progress": 0.6,
  "attempts": 1, "result_url": null, "error": null }

How it works

Leases and visibility timeouts

A queue must not lose a job if the worker that took it dies. So taking a job does not delete it; it grants a lease (temporary ownership). In Amazon SQS this is the visibility timeout: a received message is hidden from other consumers for, say, 30 seconds (the default; up to 12 hours). If the worker finishes, it deletes the message. If it crashes, the timeout expires and the message becomes visible to another worker.

A lease expires and another worker takes over

Two consequences matter:

  • Delivery is at least once. A worker that finishes the work but crashes before deleting the message will see the job run twice. Every job must therefore be idempotent (safe to run more than once), for example by writing results under a deterministic key or checking a "done" marker first. See duplicates and idempotency.
  • Long jobs must extend their lease. A job that legitimately takes 10 minutes under a 30-second timeout will be handed to a second worker while the first is still busy. The worker sends a heartbeat (SQS: ChangeMessageVisibility) every so often to extend the lease.

You don't need a separate queue product to start. Postgres can act as a job queue: FOR UPDATE SKIP LOCKED lets many workers each claim a different row without blocking each other.

UPDATE jobs
SET status = 'running', attempts = attempts + 1,
    lease_until = now() + interval '5 minutes'
WHERE id = (
  SELECT id FROM jobs
  WHERE run_at <= now()
    AND (status = 'queued' OR (status = 'running' AND lease_until < now()))
  ORDER BY run_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id, kind, payload, attempts;

The second condition picks up jobs whose worker died. The run_at column also gives you delayed jobs for free.

Retries with backoff, then a dead-letter queue

Failures fall into two kinds. Transient ones (a timeout, an HTTP 503, a deadlock) usually succeed on retry. Permanent ones (a corrupt file, a missing record, a bug) never will. Retry the first kind with exponential backoff (wait 1 s, 2 s, 4 s, 8 s...) plus jitter (randomness, so 10,000 failed jobs don't all retry in the same second). Cap the delay and the number of attempts.

Backoff, then a dead-letter queue for jobs that keep failing

A job that exhausts its attempts goes to a dead-letter queue (DLQ): a holding area for failed jobs, with an alert on its size. A human fixes the cause and replays the jobs back into the main queue. In SQS this is a redrive policy with maxReceiveCount.

import random

BASE_SECONDS, CAP_SECONDS, MAX_ATTEMPTS = 2, 600, 6

def retry_delay(attempt: int) -> float:
    # "full jitter": random point between 0 and the exponential ceiling
    return random.uniform(0, min(CAP_SECONDS, BASE_SECONDS * 2 ** attempt))

def on_failure(db, job, error, permanent: bool) -> None:
    if permanent or job.attempts >= MAX_ATTEMPTS:
        db.execute("UPDATE jobs SET status = 'dead', error = %s WHERE id = %s",
                   (str(error), job.id))
        return
    db.execute("""UPDATE jobs SET status = 'queued', lease_until = NULL,
                  run_at = now() + make_interval(secs => %s), error = %s
                  WHERE id = %s""", (retry_delay(job.attempts), str(error), job.id))

Delayed and recurring jobs

A delayed job runs at a given time: store run_at and only claim jobs whose time has come (above), or keep a Redis sorted set scored by run time and move due entries to the work queue. SQS supports delays of up to 15 minutes per message; longer delays need your own store.

A recurring job ("every night at 02:00") needs a scheduler that enqueues a job each time; the work itself still runs on ordinary workers. Run more than one scheduler for availability and two of them will fire the same slot, so make the enqueue idempotent: a unique key such as nightly-report:2026-11-19 means the second insert fails harmlessly. The distributed job scheduler case study goes deep on this.

Breaking up very long jobs

A 6-hour job that fails at hour five and restarts from zero will rarely finish. Split it: a parent job enqueues one child per chunk (per 1,000 rows, per video segment), and a counter or a final "join" job notices when all children are done. Or checkpoint: save progress every few minutes so a retry resumes rather than restarts.

Worked example: sizing the workers

The site receives 20 uploads a second at peak, and each needs on average 90 CPU-seconds of transcoding.

  • Work arriving: 20 × 90 = 1,800 CPU-seconds per second. By Little's law (work in progress = arrival rate × time per job), that is 1,800 jobs running at once.
  • At one job per core and 32 cores per machine: 1,800 ÷ 32 ≈ 57 machines, around 70 with 20% headroom.
  • Suppose peak lasts 2 hours and you deliberately run only 45 machines (1,440 cores). The shortfall is 360 CPU-seconds per second, about 4 jobs a second, so ~29,000 jobs back up by the end of the peak.
  • Off-peak, uploads drop to 10 a second (900 cores busy), leaving 540 spare cores. The backlog of 29,000 × 90 CPU-seconds clears in about 29,000 × 90 ÷ 540 ≈ 4,800 s, roughly 80 minutes.

That trade (cheaper fleet, longer waits at peak) is a product decision. Make it visible: autoscale on queue depth and age of the oldest job, not on CPU, because idle-looking workers can sit beside a growing backlog if they are blocked on I/O.

Options and trade-offs

ToolGood atWeak atUse when
Postgres table + SKIP LOCKEDNo new infrastructure; transactional enqueue with your dataThousands of jobs/s at most; polling loadSmall to medium systems
Redis-based queue (Sidekiq, BullMQ, Celery)Fast, simple, delayed jobs built inDurability depends on Redis persistence settingsTypical web app background work
Managed queue (SQS, Cloud Tasks)No servers; leases, DLQ, scaling built in15-minute max delay in SQS; vendor limitsMost cloud systems
Kafka as a job queueHuge throughput, replayOne slow message blocks its partition; no per-message retry or leaseHigh-volume event processing, not individual slow jobs
Workflow engine (Temporal, Step Functions)Multi-step jobs with durable state, timers, retries per stepAnother system to learn and runLong, multi-step business processes

One subtle point with a separate queue: if you write the job to the queue and the related row to your database as two steps, a crash between them leaves one without the other. The transactional outbox (write the job into an outbox table in the same transaction, then relay it to the queue) closes that gap; see transactions across services.

When not to use it

  • The user needs the answer to continue. A login or a price check should be made fast, not made asynchronous.
  • The work takes 200 ms. A queue adds latency, a status endpoint and failure modes. Make the request faster instead.
  • Ordering across jobs matters strictly. Workers run in parallel and retries reorder things; if order matters, partition by key and process one key at a time.

Which case studies use it

In the interview

Your turn: a CSV export

Users of an analytics product can export up to 50 million rows as a CSV. Exports take from 10 seconds to 40 minutes. Design the flow from the button click to the user downloading the file, including what happens if a worker dies at minute 30.

Compare with our answer
  • POST /exports writes a job row and returns 202 + export ID; the UI polls GET /exports/{id} (or the user gets an email with a link).
  • Workers stream rows in chunks (e.g. by primary-key range) to object storage using a multipart upload, saving the last finished chunk as a checkpoint.
  • A lease with heartbeats, so a dead worker's job is picked up after the lease expires and resumes from the checkpoint, not row zero.
  • The finished file is served with a pre-signed URL that expires; old exports are deleted by a lifecycle rule.
  • Limits: one running export per user, a separate queue so huge exports don't delay small ones.

A strong answer mentions: 202 + job ID with polling or email · chunking and checkpoints · lease expiry and heartbeats · idempotent chunk writes · a separate queue or priority for big jobs · pre-signed download URL.

Your turn: spot the bug

A team sends welcome emails from SQS workers with a 30-second visibility timeout. The email provider sometimes takes 45 seconds to respond. Customers complain about receiving two or three identical welcome emails. Why, and what would you change?

Compare with our answer

The provider call outlives the 30-second lease. The message becomes visible again while the first worker is still waiting, a second worker takes it and sends again; sometimes a third does too. The first worker then deletes the message, but the duplicates have already gone out.

A strong answer mentions:

  • The visibility timeout is shorter than the job, so at-least-once delivery turns into routine duplicates.
  • Raise the timeout well above the worst case, or extend it with a heartbeat while waiting.
  • Make sending idempotent: record "welcome sent for user X" (or pass an idempotency key the provider honours) and check it before sending.
  • Set a client timeout on the provider call shorter than the lease.
  • Add metrics on receive count > 1 to catch it early.
Background jobs on one page
  • Trigger: work over ~1–2 s, bursty load, retries needed, or something scheduled for later.
  • Shape: validate → store input → enqueue job → 202 Accepted + job ID → workers → status via polling, push or webhook.
  • Queues are at-least-once: jobs must be idempotent.
  • Lease / visibility timeout (SQS default 30 s, max 12 h); heartbeat to extend; fencing token against a stale worker.
  • Retry transient errors with exponential backoff + jitter, cap attempts (~5–8), then a DLQ with an alert and replay.
  • Postgres FOR UPDATE SKIP LOCKED is a fine queue at modest scale; run_at gives delays.
  • Recurring jobs: scheduler enqueues with a unique key per slot; workers do the work.
  • Capacity: jobs in flight = rate × duration; autoscale on queue age, separate queues per priority.
Check what stuck

1. An SQS worker resizes images in about 5 seconds, but sometimes the same image is processed twice and two thumbnails appear. What is the most likely cause?

2. A report job takes up to 20 minutes, but the queue's visibility timeout is 5 minutes. What should change?

3. A payment provider has an outage for 10 minutes. 40,000 jobs that call it fail and are retried every 5 seconds. When the provider recovers, it falls over again. What would have prevented this?

4. Three scheduler instances run for high availability, and the nightly billing job is enqueued three times. What is the cleanest fix?

5. A shop runs 300 bulk-import jobs during the day, each taking 30 minutes. Password-reset emails, processed by the same workers, now take 25 minutes to arrive. What is the best fix?

6. Uploads arrive at 50 per second and each takes 12 seconds of single-core work. Roughly how many cores must be busy at peak to keep up?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.