Synthetic checks

Probe an endpoint from the platform every few minutes and open an incident when it stops answering; checks suggested from the service graph run in trial first.

How-to · Updated 2026-10-26 · Describes release 0.1.164

A synthetic check is an HTTP(S) request or a TCP connect the platform makes to one of your endpoints on a fixed interval. It catches an outage a user would see — a load balancer answering 503, a site that stopped listening — even where no metric or alarm covers it. No model decides anything here: a check that fails sends an ordinary alert (source synthetic) into the same pipeline as your own alerts, and its incident is investigated like any other.

Where it lives: Settings · Data in → Synthetic checks (in the product: /portal/settings/monitors).

Before you start

Add a check

  1. Open Synthetic checks and fill Add check: a Name, the Kind (HTTP(S) request or TCP connect) and the Target — an http:// or https:// URL, or host:port for TCP. A URL may not carry a user name or password.
  2. For HTTP choose the Method (GET or HEAD) and the Expected status — one code or a range, 200-399 by default. A redirect is not followed: a 302 is the answer. With GET you may add Text the page must contain; it is looked for in the first 64 KB and the body is never stored.
  3. Set Every (seconds) (60 to 600, 300 by default), the Service the incident opens on (the target's host when empty) and the Severity (critical by default).
  4. Select Add check. You should see the check under Checks with Not checked yet — the first run starts within a minute, then Healthy — answered HTTP 200 in … ms after two passing runs.

Run now runs a check at once and records it like a scheduled run.

What a failure does

A failed run (a status outside the expected range, no answer within 10 seconds, the text missing) is tried once more 10 seconds later; only a failure that repeats counts, so a blip that recovers within it raises nothing. A confirmed failure turns the check Unhealthy at once and opens (or joins) an incident on its service with the alert Synthetic check <name>; the check's row links it. Two passing runs in a row make it Healthy again and resolve the alert. Disabling or deleting a check while it is unhealthy resolves its alert with check removed. The alert carries the target without its query string, the status code, the latency and where it was probed from.

You should see an incident whose signal reads Synthetic check <name> failed with the source synthetic, and on recovery the alert resolved.

Suggestions from the service graph

Once a day (and when you select Suggest now) the platform suggests a check for every internet-facing application load balancer — its DNS name on its HTTPS listener, else HTTP — and every API Gateway API (its execute-api host), at most 20 — in a workspace with workspace scope tags, only the resources carrying all of them (an untagged one is outside the scope), and of a resource harvested through an AWS connection with Only findings on resources tagged, only one carrying that tag — so another team's load balancer on a shared account is never suggested or probed. A suggestion nobody kept whose resource falls outside the scope is removed at the next daily run, also while the daily suggestions are off; a kept one stays. Any answer below 500 counts as up: a 403 or 404 at / is the endpoint answering. A suggestion probes the load balancer's own DNS name (*.elb.amazonaws.com), which the certificate of an HTTPS listener does not name (it names your domain): the certificate chain is still verified, the name is not, so a suggestion never fails on the name alone. A check you add yourself verifies the name too, and so does a suggestion whose target you change to another name. A suggestion runs in Trial under Suggested from the service graph (trial): it counts the failures it would have alerted on (Would have alerted 2 times so far) and opens nothing. Keep arms it — from then on it alerts like your own checks, at warning severity until you change it. Discard removes it for good: it is not suggested again, and its name is free for a check of your own. Turn the daily suggestions off with the workspace setting aiops_monitor_derive (on by default).

Synthetic checks in post-change validation

When a fix's change request lists a service that an armed check names as its Service, the post-change validation rounds read that check too: its latest run in each round — or after it, until the round is read — passes or fails the round, and a run one round counted never counts for the next. The rounds never start a probe of their own.

Limits

At most 100 checks per workspace, 20 of them suggestions. The platform runs at most 50 due checks a minute, 10 at a time and at most 3 of one workspace's, so one workspace's slow targets never hold up the others' checks. Checks of a private network need an agent in it; that lane is not in this release, and such a check stays not checked.

Next steps