Monitoring and alerts
Metrics, logs and traces; what to measure (golden signals, RED, USE); why percentiles beat averages; SLOs and error budgets; and alerts that wake people for the right reasons.
On this page
The problem: your users are your monitoring
It is 2 a.m. A database disk fills up, checkout starts failing, and nobody on the team knows. The first signal is a flood of angry posts the next morning. By then you have lost hours of orders, and you have no record of what happened, when it started, or why.
Without monitoring, users find the outage first
- You can’t fix what you can’t see. Without monitoring, the first sign of trouble is angry users. With it, you know before they do.
Monitoring means collecting signals from your running system so you can answer three questions quickly: is it working for users right now?, if not, where is it broken?, and is it getting worse? Alerting is the part that pages a human when the answer to the first question is "no". You will hear the word observability too: the ability to ask new questions of your system from the data it emits, without shipping new code first.
Three kinds of signal
Metrics, logs and traces answer different questions
- Monitoring has three kinds of signal. Metrics are numbers over time, like requests per second. Logs are records of individual events. Traces follow one request across every service it touches.
| Signal | What it is | Good at | Weak at | Typical tools |
|---|---|---|---|---|
| Metrics | Numbers sampled over time: requests per second, error count, p99 latency, CPU | Cheap to store and query; dashboards and alerts; spotting trends | No detail about any single request | Prometheus, Grafana, Datadog, CloudWatch |
| Logs | One record per event: payment 88 failed: card declined | Full detail; the "what exactly happened" | Expensive at volume; hard to aggregate | Elasticsearch/OpenSearch, Loki, Splunk |
| Traces | One request's path across services, as timed spans (one span per step) | Finding which service made a request slow | Usually sampled; needs every service to pass the trace ID along | Jaeger, Tempo, Zipkin, via OpenTelemetry |
They work together. A metric tells you error rate jumped at 02:14. A trace of a failing request shows the time went into the payments database. The logs of that database show "disk full". OpenTelemetry is the vendor-neutral standard most teams now use to emit all three from application code.
What to measure
You could measure thousands of things. Three short checklists keep you focused.
The four golden signals
- Google’s engineers suggest four golden signals for every service: latency, how slow; traffic, how busy; errors, how often it fails; and saturation, how full it is: CPU, memory, disk.
| Checklist | Measures | Apply it to |
|---|---|---|
| Golden signals (Google SRE book) | Latency, traffic, errors, saturation | Any user-facing service |
| RED | Rate (requests/s), Errors (failed requests/s), Duration (latency distribution) | Request-driven services: APIs, microservices |
| USE | Utilisation (% busy), Saturation (queued work), Errors | Resources: CPU, disks, network, connection pools, queues |
RED tells you what users feel; USE tells you which resource is the cause. Saturation is the subtle one: a CPU at 70% utilisation can look fine while 200 requests wait in a thread pool queue. Queue depth, connection-pool wait time and consumer lag are saturation signals, and they usually rise before latency does.
Most metric systems give you four types: a counter only goes up (total requests; you graph its rate), a gauge goes up and down (queue length, memory in use), a histogram counts values into buckets (latency between 0–50 ms, 50–100 ms…), and a summary computes percentiles on each host.
Averages lie: use percentiles
The average looks healthy while 2% of users wait 3 seconds
A percentile says "X% of requests were faster than this". The p50 (median) is the typical request; p99 is the slowest 1 in 100. Averages hide the slow tail because a few very slow requests barely move them, yet those slow requests are real users, and often your biggest customers (the ones with the most data).
The tail matters more as systems fan out. If a page calls 100 backend services in parallel and each is slow 1% of the time, the page is slow whenever any of them is:
1 − 0.99^100 ≈ 63% of page loads hit at least one slow call.
So a backend's p99 becomes the page's typical experience. This is why serious teams track p99 and p99.9 for internal services, not only p50.
SLIs, SLOs, SLAs and error budgets
Three terms that are easy to mix up:
- SLI (service level indicator): a measured number about user experience. "Share of checkout requests that succeed in under 500 ms."
- SLO (service level objective): your internal target for an SLI over a window. "99.9% over 30 days."
- SLA (service level agreement): a contract with customers, with money attached if you miss it. Set it looser than your SLO (say 99.5%), so you notice trouble before it costs you.
Worked example: what the nines allow
A 30-day window has 43,200 minutes. The allowed failure is the error budget:
| SLO | Allowed failure | Budget per 30 days (as full outage) |
|---|---|---|
| 99% | 1% | 7.2 hours |
| 99.9% | 0.1% | 43 minutes |
| 99.95% | 0.05% | 22 minutes |
| 99.99% | 0.01% | 4.3 minutes |
The error budget is a shared allowance for risk
The budget does not have to be spent as one outage: 0.1% of requests failing all month spends it too. It turns reliability into a decision tool. While budget remains, ship features and run experiments. When it is nearly gone, slow releases and spend the time on reliability. 100% is never the target: users cannot tell 99.99% from 100% through their own flaky Wi-Fi, and every extra nine costs roughly ten times more.
Burn-rate alerts
"Error rate above 0.1%" fires on every short blip. A better alert asks how fast you are spending the budget. Burn rate is the error rate divided by the allowed rate: burning at 1× uses exactly the budget in 30 days; at 14.4× you would use it in about 2 days. The Google SRE Workbook suggests: page if the burn rate is above 14.4× over the last hour (2% of the monthly budget gone) and over the last 5 minutes (so it is still happening); open a ticket if it is above 1× over 3 days. The two windows catch fast disasters quickly and slow leaks eventually, without paging for a 30-second hiccup.
Alerts that people trust
An alert is a threshold on a signal, and thresholds are a balancing act
- A dashboard shows these at a glance. An alert watches one of them and wakes someone up when it crosses a threshold, like error rate above 5% for 2 minutes.
- Thresholds are a balancing act. Too sensitive, and alerts fire for every harmless blip; people learn to ignore them, called alert fatigue. Too relaxed, and you miss real outages.
Good alerting follows a few rules:
- Page on symptoms, not causes. "Checkout success rate below SLO" is a symptom users feel. "CPU at 85%" is a cause that may be harmless. Show causes on dashboards; page only when users are hurt or about to be.
- Every page must be actionable. If the on-call engineer cannot do anything, it should be a ticket or a dashboard line, not a 3 a.m. phone call.
- Require duration. "Above threshold for 5 minutes" filters spikes.
- Link a runbook: a short page explaining what the alert means and the first steps to take.
- Count your pages. More than a couple per on-call shift means people will start ignoring them: alert fatigue.
Monitor from outside too. Synthetic checks (a script that logs in and buys a test item every minute from another region) catch failures your own metrics miss, such as DNS or certificate problems.
Finding where the time went: tracing
A trace shows which span ate the time
- When an alert fires, traces show where the time went. One slow request might spend 20 milliseconds in the app, but 900 in the payments database: now you know where to look.
When a request enters the system, the first service creates a trace ID and passes it on in a header (the W3C traceparent standard) to every service it calls. Each service records a span with start time, duration and the trace ID, and a collector stitches the spans into a timeline.
Tracing every request is expensive, so traces are sampled. Head sampling decides at the start (keep 1%), which is cheap but usually misses the rare slow or failing request you care about. Tail sampling buffers spans and decides after the request finishes (keep all errors and everything over 1 s, plus 1% of the rest), which is more useful and costs more memory in the collector.
When not to go further
- A small product needs RED metrics per endpoint, structured logs, an uptime check and two or three SLO alerts. Distributed tracing across one service adds little.
- Do not build your own metrics database or tracing system; use Prometheus/Grafana, a hosted service, and OpenTelemetry.
- Do not log everything at full volume forever. Logs at 50,000 requests/s × 1 KB is about 4 TB a day. Sample success logs, keep error logs, and set retention by value.
In the interview
A checkout API calls Cart, Inventory and Payments, and writes orders to Postgres. It handles 2,000 requests/s at peak. Choose the SLIs, the SLO, the alerts and the dashboards.
Compare with our answer
A sensible plan: SLI = share of checkout requests that return success (not 5xx) in under 1 s; SLO = 99.9% over 30 days (about 43 minutes of budget). Page on burn rate (fast and slow windows). Dashboards: RED for checkout and each dependency, Postgres connection-pool wait, replication lag, Payments error codes split by "declined" (user's problem) versus "timeout" (ours). Trace every request with tail sampling that keeps all errors and slow requests. Run a synthetic test purchase every minute from another region.
A strong answer mentions: an SLI defined from the user's view · an SLO with its error budget in minutes · burn-rate alerts rather than raw thresholds · RED per dependency and USE for the database pool · separating user errors from system errors · tracing or synthetic checks.
A team's alert rules: (1) page when average latency is above 500 ms for 1 minute; (2) page when any server's CPU is above 80%; (3) a metric request_latency{user_id, route}; (4) the fleet p99 dashboard averages each server's p99. Their on-call gets 30 pages a week and still missed last month's outage. What would you change?
Compare with our answer
- (1) The average hides the tail; use p99 from histograms, and a longer window or a burn-rate rule to avoid paging on blips.
- (2) CPU is a cause, not a symptom; 80% CPU may be perfectly healthy. Put it on a dashboard and page on user-facing SLOs.
- (3)
user_idis unbounded and will explode the number of time series; move it to logs and traces. - (4) Averaging percentiles is mathematically wrong; sum histogram buckets across servers, then compute p99.
- 30 pages a week means alert fatigue; delete or downgrade every alert that is not actionable, and add runbooks.
A strong answer mentions: percentiles over averages · symptoms over causes · bounded label cardinality · merging histograms, not percentiles · cutting non-actionable pages.
- Metrics (cheap numbers over time), logs (per-event detail), traces (one request across services). Emit all three with OpenTelemetry; link them with a trace ID.
- Golden signals: latency, traffic, errors, saturation. RED for services; USE for resources. Saturation (queues, pool waits) warns first.
- Use p50/p99 from histograms; never average percentiles. Fan-out: 100 calls at 1% slow each → 63% of pages slow.
- SLI = measurement; SLO = target (99.9% / 30 days ≈ 43 min budget); SLA = contract, looser than the SLO.
- Error budget left → ship; budget gone → fix reliability.
- Page on symptoms with burn-rate alerts (14.4× over 1 h and 5 min); every page actionable with a runbook.
- Keep metric labels bounded; high-cardinality detail goes to logs and traces.
- Monitor the monitor: separate failure domain, external checks, dead man's switch.
1. Your API's average latency is 120 ms and nobody complains in aggregate, but your biggest customer says the app is "constantly slow". What should you look at first?
Averages hide the tail, and heavy users often land in it (more data, more calls per page). Percentiles show it. CPU and traffic are possible causes to check later, but first confirm the symptom.
2. A service has an SLO of 99.9% successful requests over 30 days. Roughly how much full-outage time does its error budget allow per window?
30 days is 43,200 minutes, and 0.1% of that is about 43 minutes. 4.3 minutes is 99.99%, 22 minutes is 99.95%, and 7.2 hours is 99%.
3. Server A reports a p99 of 100 ms and server B a p99 of 900 ms. How do you get the true p99 across both?
Percentiles do not combine by any kind of averaging; you need the underlying distribution. Summing histogram buckets gives it. The fleet p99 can be lower than the worst server's if that server handles few requests.
4. Which alert is the best one to page an on-call engineer at night?
Paging alerts should be symptoms users feel and need action now. A fast error-budget burn is both. High CPU may be harmless, the disk trend can be a daytime ticket, and autoscaling is normal behaviour.
5. A team adds a customer_id label to their request-latency metric to debug one customer. Prometheus memory use explodes. Why?
Cardinality is the number of series, and an unbounded label multiplies it. Per-customer detail belongs in logs or traces, which are built for high-cardinality fields.
6. A page makes 50 parallel calls to backends, and each call is independently slow 2% of the time. Roughly what share of page loads include at least one slow call?
The page is fast only if all 50 calls are fast: 0.98^50 ≈ 0.36, so about 64% are slow. This is why backend tail latency matters so much in fan-out systems.
7. You trace 1% of requests chosen at random when they start, but the traces never contain the rare timeouts you are investigating. What change helps most?
Head sampling decides before anyone knows a request will fail, so rare failures are mostly dropped. Tail sampling keeps the interesting ones. A trace ID as a metric label would also cause a cardinality explosion.
Score 60% or more and the lesson is marked complete.