Your alerts don't know what Tuesday 3am looks like

At 3:12 in the morning the pager went off, as it did most nights: CPU above 80 percent for five minutes. The on-call engineer acknowledged it from bed without fully waking, because it was the nightly batch window, the same batch window as yesterday, running the same load it always runs. The runbook for that page was effectively one line: acknowledge it. Two weeks later, checkout latency drifted from 300 to 900 milliseconds across four days, quietly degrading conversion the whole way, and no alert fired at all, because 900 milliseconds was still under the threshold someone had picked in a config review eighteen months earlier.

The alerting system woke a human 26 times that month for things that were normal and stayed silent for the one thing that was not. Both failures have invoices. The false pages bill in engineers: pager fatigue is a documented driver of on-call attrition. The silent incident bills in customers, and it billed for four days.

An alert that fires every night is not monitoring the system. It is training your best people to ignore alarms.

The root defect is that a static threshold encodes a single guess about "normal" for a system whose normal is a shape, not a level. Production metrics breathe: daily cycles, weekday-weekend swings, batch windows, deploy spikes, end-of-month load. Eighty percent CPU at 3am during the batch run is healthy; the same reading at 2pm on a Tuesday is a fire. Fifty requests per second is normal at noon and, for some businesses, an outage at midnight. One number cannot be right about both, so teams pick thresholds loose enough to survive the peaks, and then drift, slow leaks, and gradual degradations glide underneath forever.

There is also a failure class thresholds cannot express at all: absence. The night the batch job silently fails to start, CPU stays low, every threshold is comfortable, and the alerting system reads the missing workload as an unusually good night. Something did not happen, and nothing you own can page on a thing that did not happen.

The method has two layers, and specialist teams run both.

The first is learned baselines. Model each key metric's expected value as a function of time, seasonal decomposition that captures the daily and weekly shape, computed robustly so past incidents do not contaminate the definition of normal. Alert on departure from the expected band, not on crossing a fixed line. The same 80 percent that is quiet during the batch window pages instantly at 2pm. The missing batch spike becomes visible, because the baseline expected load that never arrived; absence turns into a first-class signal. Deviation scoring also grades severity honestly: three percent above expected is a note in a channel, forty percent is a page.

The second layer is burn-rate alerting on service objectives, which anchors paging to user impact rather than machine internals. Define the objective, say 99.5 percent of checkouts under 500 milliseconds, and page on how fast the error budget is burning, evaluated over a fast window and a slow one together. The fast window catches sudden breakage in minutes; the slow one catches exactly the four-day drift that slid under the static threshold, because a slow leak still drains the budget. Machine-level baselines tell you something is odd; burn rates tell you customers are feeling it. The pairing is the point.

Then treat alert quality as a product: track what fraction of pages led to action, review the worst offenders monthly, and delete freely. We rebuilt alerting this way for a data platform whose on-call rotation had become a hazing ritual; pages dropped by roughly two thirds inside a quarter, and the next real incident was caught by a slow-burn alert at breakfast, hours before any threshold would have blinked.

Deleting a useless alert improves detection. Silence you can trust is the actual product.

The honest cost: learned baselines need history to learn, misbehave around genuine regime changes like a big launch, and add machinery a five-service startup does not need yet. Static thresholds are fine at small scale. The transition point is the month the on-call channel becomes something people mute.

Two numbers from your incident tooling: how many pages fired, and how many led to a human doing something. If the ratio embarrasses you, your team already knows, they are the ones acknowledging from bed. TensorLabs builds this detection layer, seasonal baselines, burn-rate objectives, the alertquality loop, on top of the observability stack you already have. Write to us with those two numbers and what woke someone last night. The gap between them is the roadmap.