How multi-region confirmation stops false alarms

Ask anyone who has carried a pager what they hate most about uptime monitoring, and you’ll hear the same answer: the alert that wasn’t real. The 3 a.m. page for a site that was fine. The Slack channel full of “down” and “up” messages thirty seconds apart. Over time, teams learn to wait before reacting, which quietly defeats the point of monitoring.

Most false alarms have nothing to do with your service. They come from the path between the probe and your service. This post is about that path, and about the one design decision that removes most of the noise.

Where false positives actually come from

A probe in Frankfurt requesting https://api.example.com/health depends on a lot of things that aren’t your API:

• The probe’s own resolver. If the datacenter’s DNS resolver hiccups, the check fails with name not resolved even though your DNS is fine and every user on earth can reach you.
• Transit between the probe and you. A routing change, a congested peering link, or a DDoS mitigation kicking in somewhere on the path produces timeouts and resets that only that region sees.
• The probe’s host. Cloud instances get live-migrated, throttled, or briefly lose network. The check runs late or not at all.
• Rate limiting and WAFs. Your edge decides the probe looks like a bot and returns a 403, from one region, because the rule is regional.
• Genuinely regional outages. A CDN edge or a cloud region has a bad day. Users in that region are affected; everyone else isn’t. Whether this deserves a page depends on your business, but it isn’t “the site is down.”

None of these are hypothetical. Each of them produces a check result that looks exactly like an outage: the request failed, the timings show where, and a naive monitor turns red.

The fix: require a second opinion

The insight is simple. Every cause above is local to one vantage point. A real outage is not. So before a monitor changes state, ask a second region.

In StatusCheck, a failed check from one region puts the monitor into a pending state. Nothing is sent. At the next scheduled check from a different region, usually within one interval, one of two things happens:

1. The second region also fails. The failure is confirmed. The monitor goes down, an incident opens, alerts go out, the status page updates.
2. The second region succeeds. The first failure is recorded as a single-region failure, visible in the monitor’s history, but the monitor stays up and nobody is paged.

That’s it. No machine learning, no “anomaly score”, no tuning. Two independent network paths agreeing is a much stronger signal than one path failing, and it costs at most one check interval of detection time.

What it costs

The honest trade-off is latency. If your monitor runs every 60 seconds from five regions, checks are staggered so that one region runs roughly every 12 seconds. A confirmed failure therefore takes between 12 and 60 seconds longer to alert than a single-region monitor would. For a 30-second interval it’s a few seconds.

We think that’s the right trade. An alert that arrives 20 seconds later but is right every time is worth more than one that arrives instantly and is wrong once a week, because the wrong ones train people to ignore the right ones.

If you want the earliest possible signal and confirmation, run the monitor from more regions at a shorter interval. Confirmation gets faster as the gap between regional checks shrinks.

What confirmation doesn’t fix

Confirmation removes noise from the probe side. It doesn’t remove noise from your side:

• A health endpoint that returns 200 while the database is unreachable will look up from every region. Assert on the body, and make the endpoint check its dependencies.
• A monitor with an aggressive timeout on a slow-by-design endpoint will fail everywhere. Set a response-time threshold and let the monitor go degraded instead of down.
• A deploy that briefly returns 502 from every region is an outage, however short. If you don’t want to be paged for a 15-second deploy blip, raise the monitor’s “consecutive failures before alerting” setting, knowing that it also delays real alerts by that many intervals.

Recommendations

• Run every customer-facing monitor from at least three regions on different continents. Two is the minimum for confirmation; three means a single region’s outage never blocks confirmation either.
• Use 60-second intervals for anything customers notice, 30 seconds for the endpoints that pay the bills.
• Look at the single-region failure history occasionally. A region that fails alone every day at the same time is telling you something about your edge configuration.
• Prefer degraded thresholds over tight timeouts for slow endpoints.

StatusCheck does the confirmation step on every plan, including Free, because we don’t think “alerts you can trust” should be a paid feature. The concepts page in the docs covers the mechanics in a little more detail.

monitoringalertingreliability