I counted 1,247 alerts in one week across a mid-size enterprise’s monitoring stack last month. Of those, 31 required actual human intervention. The rest were noise—informational, duplicate, or triggered by thresholds set three years ago for an infrastructure that no longer exists.

That’s a signal-to-noise ratio of 2.5%. And the team responsible for responding? They’d stopped trusting the alerts entirely. Critical incidents were getting lost in the flood because engineers had been conditioned to ignore everything.

This is the observability paradox. We’ve instrumented everything, shipped logs and metrics to expensive platforms, and built dashboards that look impressive in board presentations. But the operational reality is that more data has produced worse outcomes.

How Alert Fatigue Sets In

It’s a gradual process. You deploy a new monitoring tool—maybe Datadog, maybe PagerDuty on top of Prometheus, maybe Splunk’s alerting engine. The initial setup includes sensible defaults. CPU above 90% for five minutes? Alert. Disk at 85%? Alert. Response time above 500ms? Alert.

Then things change. Someone adds alerts for a new service. Another team duplicates alerts across tools because they don’t trust the primary system. A contractor sets thresholds too low because they’d rather over-alert than miss something during their engagement. Nobody cleans up alerts for decommissioned services.

Within 18 months, your alerting config is a mess. Engineers develop coping mechanisms—most commonly, they stop looking. According to PagerDuty’s 2025 State of Digital Operations report, 49% of on-call engineers reported ignoring alerts at least once per shift. That number should terrify every CTO reading this.

The Real Cost

Alert fatigue doesn’t just burn out engineers. It directly impacts your SLA performance and incident response times.

When your team gets paged 30 times in a night, the response to page 31—which might be a genuine production outage—is slower and less focused. Mean time to acknowledge (MTTA) and mean time to resolve (MTTR) both degrade. I’ve seen organisations where MTTR doubled over 12 months despite no increase in actual incident volume. The incidents weren’t getting harder to fix. People were just slower to engage because they’d been woken up for nonsense too many times.

The financial impact is straightforward to calculate. Every unnecessary alert that triggers an on-call response costs roughly $50-100 in direct labour. Multiply that by hundreds or thousands of false positives per month, and you’re looking at a significant hidden cost. That’s before accounting for the retention risk—on-call burnout is one of the top three reasons infrastructure engineers leave, according to the 2025 Stack Overflow Developer Survey.

Fixing It

The fix isn’t buying another tool. It’s doing the unglamorous work of reviewing and rationalising what you already have.

Start with an alert audit. Pull every alert that fired in the last 90 days. Categorise them: actionable (someone did something), informational (no action taken), and noise (acknowledged and closed without investigation). Anything in the noise category gets deleted or downgraded to a log entry.

Implement alert routing tiers. Not every alert needs to page a human at 3am. Build three tiers: P1 alerts that wake people up (should be rare—fewer than five per week), P2 alerts that go to a Slack channel for next-business-day review, and P3 alerts that feed into weekly reporting dashboards but don’t notify anyone.

Set dynamic thresholds. Static thresholds are the root cause of most false positives. CPU at 92% on a batch processing server at 2am on a Sunday is normal. CPU at 92% on your payment gateway during business hours is worth investigating. Modern observability platforms support anomaly detection—use it instead of hard-coded numbers.

Create ownership. Every alert should have a clear owner. If nobody can explain why an alert exists or what the response procedure is, that alert gets deleted. I worked on this exact exercise with a client alongside Team400, and we eliminated 73% of their alerts in a single sprint. Incident response times improved by 40% in the following quarter because the remaining alerts actually meant something.

Review quarterly. Alert configurations aren’t a set-and-forget exercise. Infrastructure changes, traffic patterns shift, and new services get deployed. Schedule quarterly alert reviews the same way you schedule penetration tests—as a mandatory operational hygiene task.

The Cultural Shift

The hardest part isn’t technical. It’s convincing leaders that fewer alerts is better. There’s a psychological comfort in monitoring everything—it feels responsible and thorough. But monitoring without curation is just noise generation.

Your observability stack should be a trusted colleague who only interrupts you when something genuinely needs attention. If your team doesn’t trust their alerts, you don’t have an observability problem. You have an operational maturity problem. And the only way to fix it is to sit down, do the boring review work, and start deleting things.

The engineers who are currently ignoring your alerts will thank you.