Staying up · Chapter 16

Monitoring and alerts

Metrics, logs and traces; what to measure (golden signals, RED, USE); why percentiles beat averages; SLOs and error budgets; and alerts that wake people for the right reasons.

IntermediateInterview priority: Medium35 min
On this page
  1. The problem: your users are your monitoring
  2. Three kinds of signal
  3. What to measure
  4. Averages lie: use percentiles
  5. SLIs, SLOs, SLAs and error budgets
  6. Alerts that people trust
  7. Finding where the time went: tracing
  8. When not to go further
  9. In the interview
  10. Quiz

The problem: your users are your monitoring

It is 2 a.m. A database disk fills up, checkout starts failing, and nobody on the team knows. The first signal is a flood of angry posts the next morning. By then you have lost hours of orders, and you have no record of what happened, when it started, or why.

Without monitoring, users find the outage first

  1. You can’t fix what you can’t see. Without monitoring, the first sign of trouble is angry users. With it, you know before they do.

Monitoring means collecting signals from your running system so you can answer three questions quickly: is it working for users right now?, if not, where is it broken?, and is it getting worse? Alerting is the part that pages a human when the answer to the first question is "no". You will hear the word observability too: the ability to ask new questions of your system from the data it emits, without shipping new code first.

Three kinds of signal

Metrics, logs and traces answer different questions

  1. Monitoring has three kinds of signal. Metrics are numbers over time, like requests per second. Logs are records of individual events. Traces follow one request across every service it touches.
SignalWhat it isGood atWeak atTypical tools
MetricsNumbers sampled over time: requests per second, error count, p99 latency, CPUCheap to store and query; dashboards and alerts; spotting trendsNo detail about any single requestPrometheus, Grafana, Datadog, CloudWatch
LogsOne record per event: payment 88 failed: card declinedFull detail; the "what exactly happened"Expensive at volume; hard to aggregateElasticsearch/OpenSearch, Loki, Splunk
TracesOne request's path across services, as timed spans (one span per step)Finding which service made a request slowUsually sampled; needs every service to pass the trace ID alongJaeger, Tempo, Zipkin, via OpenTelemetry

They work together. A metric tells you error rate jumped at 02:14. A trace of a failing request shows the time went into the payments database. The logs of that database show "disk full". OpenTelemetry is the vendor-neutral standard most teams now use to emit all three from application code.

What to measure

You could measure thousands of things. Three short checklists keep you focused.

The four golden signals

  1. Google’s engineers suggest four golden signals for every service: latency, how slow; traffic, how busy; errors, how often it fails; and saturation, how full it is: CPU, memory, disk.
ChecklistMeasuresApply it to
Golden signals (Google SRE book)Latency, traffic, errors, saturationAny user-facing service
REDRate (requests/s), Errors (failed requests/s), Duration (latency distribution)Request-driven services: APIs, microservices
USEUtilisation (% busy), Saturation (queued work), ErrorsResources: CPU, disks, network, connection pools, queues

RED tells you what users feel; USE tells you which resource is the cause. Saturation is the subtle one: a CPU at 70% utilisation can look fine while 200 requests wait in a thread pool queue. Queue depth, connection-pool wait time and consumer lag are saturation signals, and they usually rise before latency does.

Most metric systems give you four types: a counter only goes up (total requests; you graph its rate), a gauge goes up and down (queue length, memory in use), a histogram counts values into buckets (latency between 0–50 ms, 50–100 ms…), and a summary computes percentiles on each host.

Averages lie: use percentiles

The average looks healthy while 2% of users wait 3 seconds

A percentile says "X% of requests were faster than this". The p50 (median) is the typical request; p99 is the slowest 1 in 100. Averages hide the slow tail because a few very slow requests barely move them, yet those slow requests are real users, and often your biggest customers (the ones with the most data).

The tail matters more as systems fan out. If a page calls 100 backend services in parallel and each is slow 1% of the time, the page is slow whenever any of them is:

1 − 0.99^100 ≈ 63% of page loads hit at least one slow call.

So a backend's p99 becomes the page's typical experience. This is why serious teams track p99 and p99.9 for internal services, not only p50.

SLIs, SLOs, SLAs and error budgets

Three terms that are easy to mix up:

  • SLI (service level indicator): a measured number about user experience. "Share of checkout requests that succeed in under 500 ms."
  • SLO (service level objective): your internal target for an SLI over a window. "99.9% over 30 days."
  • SLA (service level agreement): a contract with customers, with money attached if you miss it. Set it looser than your SLO (say 99.5%), so you notice trouble before it costs you.

Worked example: what the nines allow

A 30-day window has 43,200 minutes. The allowed failure is the error budget:

SLOAllowed failureBudget per 30 days (as full outage)
99%1%7.2 hours
99.9%0.1%43 minutes
99.95%0.05%22 minutes
99.99%0.01%4.3 minutes

The error budget is a shared allowance for risk

The budget does not have to be spent as one outage: 0.1% of requests failing all month spends it too. It turns reliability into a decision tool. While budget remains, ship features and run experiments. When it is nearly gone, slow releases and spend the time on reliability. 100% is never the target: users cannot tell 99.99% from 100% through their own flaky Wi-Fi, and every extra nine costs roughly ten times more.

Burn-rate alerts

"Error rate above 0.1%" fires on every short blip. A better alert asks how fast you are spending the budget. Burn rate is the error rate divided by the allowed rate: burning at 1× uses exactly the budget in 30 days; at 14.4× you would use it in about 2 days. The Google SRE Workbook suggests: page if the burn rate is above 14.4× over the last hour (2% of the monthly budget gone) and over the last 5 minutes (so it is still happening); open a ticket if it is above 1× over 3 days. The two windows catch fast disasters quickly and slow leaks eventually, without paging for a 30-second hiccup.

Alerts that people trust

An alert is a threshold on a signal, and thresholds are a balancing act

  1. A dashboard shows these at a glance. An alert watches one of them and wakes someone up when it crosses a threshold, like error rate above 5% for 2 minutes.
  2. Thresholds are a balancing act. Too sensitive, and alerts fire for every harmless blip; people learn to ignore them, called alert fatigue. Too relaxed, and you miss real outages.

Good alerting follows a few rules:

  1. Page on symptoms, not causes. "Checkout success rate below SLO" is a symptom users feel. "CPU at 85%" is a cause that may be harmless. Show causes on dashboards; page only when users are hurt or about to be.
  2. Every page must be actionable. If the on-call engineer cannot do anything, it should be a ticket or a dashboard line, not a 3 a.m. phone call.
  3. Require duration. "Above threshold for 5 minutes" filters spikes.
  4. Link a runbook: a short page explaining what the alert means and the first steps to take.
  5. Count your pages. More than a couple per on-call shift means people will start ignoring them: alert fatigue.

Monitor from outside too. Synthetic checks (a script that logs in and buys a test item every minute from another region) catch failures your own metrics miss, such as DNS or certificate problems.

Finding where the time went: tracing

A trace shows which span ate the time

  1. When an alert fires, traces show where the time went. One slow request might spend 20 milliseconds in the app, but 900 in the payments database: now you know where to look.

When a request enters the system, the first service creates a trace ID and passes it on in a header (the W3C traceparent standard) to every service it calls. Each service records a span with start time, duration and the trace ID, and a collector stitches the spans into a timeline.

Tracing every request is expensive, so traces are sampled. Head sampling decides at the start (keep 1%), which is cheap but usually misses the rare slow or failing request you care about. Tail sampling buffers spans and decides after the request finishes (keep all errors and everything over 1 s, plus 1% of the rest), which is more useful and costs more memory in the collector.

When not to go further

  • A small product needs RED metrics per endpoint, structured logs, an uptime check and two or three SLO alerts. Distributed tracing across one service adds little.
  • Do not build your own metrics database or tracing system; use Prometheus/Grafana, a hosted service, and OpenTelemetry.
  • Do not log everything at full volume forever. Logs at 50,000 requests/s × 1 KB is about 4 TB a day. Sample success logs, keep error logs, and set retention by value.

In the interview

Your turn: monitor a checkout service

A checkout API calls Cart, Inventory and Payments, and writes orders to Postgres. It handles 2,000 requests/s at peak. Choose the SLIs, the SLO, the alerts and the dashboards.

Compare with our answer

A sensible plan: SLI = share of checkout requests that return success (not 5xx) in under 1 s; SLO = 99.9% over 30 days (about 43 minutes of budget). Page on burn rate (fast and slow windows). Dashboards: RED for checkout and each dependency, Postgres connection-pool wait, replication lag, Payments error codes split by "declined" (user's problem) versus "timeout" (ours). Trace every request with tail sampling that keeps all errors and slow requests. Run a synthetic test purchase every minute from another region.

A strong answer mentions: an SLI defined from the user's view · an SLO with its error budget in minutes · burn-rate alerts rather than raw thresholds · RED per dependency and USE for the database pool · separating user errors from system errors · tracing or synthetic checks.

Your turn: spot the bugs

A team's alert rules: (1) page when average latency is above 500 ms for 1 minute; (2) page when any server's CPU is above 80%; (3) a metric request_latency{user_id, route}; (4) the fleet p99 dashboard averages each server's p99. Their on-call gets 30 pages a week and still missed last month's outage. What would you change?

Compare with our answer
  • (1) The average hides the tail; use p99 from histograms, and a longer window or a burn-rate rule to avoid paging on blips.
  • (2) CPU is a cause, not a symptom; 80% CPU may be perfectly healthy. Put it on a dashboard and page on user-facing SLOs.
  • (3) user_id is unbounded and will explode the number of time series; move it to logs and traces.
  • (4) Averaging percentiles is mathematically wrong; sum histogram buckets across servers, then compute p99.
  • 30 pages a week means alert fatigue; delete or downgrade every alert that is not actionable, and add runbooks.

A strong answer mentions: percentiles over averages · symptoms over causes · bounded label cardinality · merging histograms, not percentiles · cutting non-actionable pages.

Monitoring on one page
  • Metrics (cheap numbers over time), logs (per-event detail), traces (one request across services). Emit all three with OpenTelemetry; link them with a trace ID.
  • Golden signals: latency, traffic, errors, saturation. RED for services; USE for resources. Saturation (queues, pool waits) warns first.
  • Use p50/p99 from histograms; never average percentiles. Fan-out: 100 calls at 1% slow each → 63% of pages slow.
  • SLI = measurement; SLO = target (99.9% / 30 days ≈ 43 min budget); SLA = contract, looser than the SLO.
  • Error budget left → ship; budget gone → fix reliability.
  • Page on symptoms with burn-rate alerts (14.4× over 1 h and 5 min); every page actionable with a runbook.
  • Keep metric labels bounded; high-cardinality detail goes to logs and traces.
  • Monitor the monitor: separate failure domain, external checks, dead man's switch.
Interactive labPractise it: the monitoring playgroundOpens in the ByteAtlas lab. Your progress is saved in this browser.
Check what stuck

1. Your API's average latency is 120 ms and nobody complains in aggregate, but your biggest customer says the app is "constantly slow". What should you look at first?

2. A service has an SLO of 99.9% successful requests over 30 days. Roughly how much full-outage time does its error budget allow per window?

3. Server A reports a p99 of 100 ms and server B a p99 of 900 ms. How do you get the true p99 across both?

4. Which alert is the best one to page an on-call engineer at night?

5. A team adds a customer_id label to their request-latency metric to debug one customer. Prometheus memory use explodes. Why?

6. A page makes 50 parallel calls to backends, and each call is independently slow 2% of the time. Roughly what share of page loads include at least one slow call?

7. You trace 1% of requests chosen at random when they start, but the traces never contain the rare timeouts you are investigating. What change helps most?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.