# Synthetic checks

> Probe an endpoint from the platform every few minutes and open an incident when it stops answering; checks suggested from the service graph run in trial first.

_How-to · Updated 2026-10-26 · Describes release 0.1.164_

A synthetic check is an HTTP(S) request or a TCP connect the platform makes to one of your endpoints on a fixed interval. It catches an outage a user would see — a load balancer answering 503, a site that stopped listening — even where no metric or alarm covers it. No model decides anything here: a check that fails sends an ordinary alert (source `synthetic`) into the same pipeline as your own alerts, and its incident is investigated like any other.

Where it lives: **Settings · Data in → [Synthetic checks](/portal/settings/monitors) (in the product: `/portal/settings/monitors`)**.

## Before you start {#before-you-start}

- Reading the page needs `datasources.view`; adding, changing, running and removing checks needs `datasources.manage`.
- The platform probes from where it runs. A target whose name resolves to a private address (10.x, 172.16–31.x, 192.168.x and the like) cannot be reached from the platform: such a check reads **Needs an agent in that network** and stays *not checked*. A host in Guardrails → **SSRF allowlist — internal hosts (one per line)** is probed anyway, and so is any host while the workspace has turned its SSRF guard off. A cloud metadata address (169.254.169.254 and its kin) is never probed.
- The suggestions come from the estate inventory: run a harvest first ([Estate harvest and x-ray](/help/estate-harvest-and-xray.md)) so the internet-facing load balancers and API Gateway APIs are known.

## Add a check {#add-a-check}

1. Open **Synthetic checks** and fill **Add check**: a **Name**, the **Kind** (*HTTP(S) request* or *TCP connect*) and the **Target** — an `http://` or `https://` URL, or `host:port` for TCP. A URL may not carry a user name or password.
2. For HTTP choose the **Method** (GET or HEAD) and the **Expected status** — one code or a range, `200-399` by default. A redirect is not followed: a `302` is the answer. With GET you may add **Text the page must contain**; it is looked for in the first 64 KB and the body is never stored.
3. Set **Every (seconds)** (60 to 600, 300 by default), the **Service** the incident opens on (the target's host when empty) and the **Severity** (critical by default).
4. Select **Add check**. **You should see** the check under **Checks** with *Not checked yet — the first run starts within a minute*, then *Healthy — answered HTTP 200 in … ms* after two passing runs.

**Run now** runs a check at once and records it like a scheduled run.

## What a failure does {#what-a-failure-does}

A failed run (a status outside the expected range, no answer within 10 seconds, the text missing) is tried once more 10 seconds later; only a failure that repeats counts, so a blip that recovers within it raises nothing. A confirmed failure turns the check **Unhealthy** at once and opens (or joins) an incident on its service with the alert *Synthetic check <name>*; the check's row links it. Two passing runs in a row make it **Healthy** again and resolve the alert. Disabling or deleting a check while it is unhealthy resolves its alert with *check removed*. The alert carries the target without its query string, the status code, the latency and where it was probed from.

**You should see** an incident whose signal reads *Synthetic check <name> failed* with the source `synthetic`, and on recovery the alert resolved.

## Suggestions from the service graph {#suggestions}

Once a day (and when you select **Suggest now**) the platform suggests a check for every internet-facing application load balancer — its DNS name on its HTTPS listener, else HTTP — and every API Gateway API (its `execute-api` host), at most 20 — in a workspace with **workspace scope** tags, only the resources carrying all of them (an untagged one is outside the scope), and of a resource harvested through an AWS connection with **Only findings on resources tagged**, only one carrying that tag — so another team's load balancer on a shared account is never suggested or probed. A suggestion nobody kept whose resource falls outside the scope is removed at the next daily run, also while the daily suggestions are off; a kept one stays. Any answer below 500 counts as up: a `403` or `404` at `/` is the endpoint answering. A suggestion probes the load balancer's own DNS name (`*.elb.amazonaws.com`), which the certificate of an HTTPS listener does not name (it names your domain): the certificate chain is still verified, the name is not, so a suggestion never fails on the name alone. A check you add yourself verifies the name too, and so does a suggestion whose target you change to another name. A suggestion runs in **Trial** under **Suggested from the service graph (trial)**: it counts the failures it would have alerted on (*Would have alerted 2 times so far*) and opens nothing. **Keep** arms it — from then on it alerts like your own checks, at warning severity until you change it. **Discard** removes it for good: it is not suggested again, and its name is free for a check of your own. Turn the daily suggestions off with the workspace setting `aiops_monitor_derive` (on by default).

## Synthetic checks in post-change validation {#validation}

When a fix's change request lists a service that an armed check names as its **Service**, the [post-change validation rounds](/help/verify-recovery.md#recheck-and-post-close-verification) read that check too: its latest run in each round — or after it, until the round is read — passes or fails the round, and a run one round counted never counts for the next. The rounds never start a probe of their own.

## Limits {#limits}

At most 100 checks per workspace, 20 of them suggestions. The platform runs at most 50 due checks a minute, 10 at a time and at most 3 of one workspace's, so one workspace's slow targets never hold up the others' checks. Checks of a private network need an agent in it; that lane is not in this release, and such a check stays *not checked*.

## Next steps {#next-steps}

- [Built-in detection (AWS)](/help/builtin-detection.md) for the CloudWatch rules over the same inventory.
- [Verify recovery](/help/verify-recovery.md) for how a fix is validated after it runs.
- API: `GET`/`POST /api/aiops/{tenant_id}/monitors`, `PATCH`/`DELETE …/monitors/{check_id}` (`{"armed": true}` keeps a suggestion), `POST …/monitors/{check_id}/run`, `POST …/monitors/derive`; the MCP tools are `list_monitors`, `create_monitor`, `update_monitor`, `delete_monitor`, `run_monitor` and `derive_monitors`.
