Verify recovery

The recheck: scheduled rechecks, post-fix validation and post-close verification, the code-first ladder, validation by metric, and what each verdict does.

How-to · Updated 2026-10-27 · Describes release 0.1.164

A close is a claim, and a released fix is a claim too: both start a bounded, read-only recheck that records a verdict against the incident. This page is the Recheck card — the scheduled recheck, post-fix validation and post-close verification, the code-first ladder, validation by metric, the verdict table and what a Still present verdict does. Mark resolved itself is under Approve and run a fix.

Recheck and post-close verification

Incident page, L6 Validation & normalisation: the latest verdict, what validation has cost so far and Verify now, then for the fix attempt the checklist — criterion, observation and its source (alert, agent, compiled) — and each verify run with its verdict, model and cost; earlier attempts fold below.
Incident page, L6 Validation & normalisation: the latest verdict, what validation has cost so far and Verify now, then for the fix attempt the checklist — criterion, observation and its source (alert, agent, compiled) — and each verify run with its verdict, model and cost; earlier attempts fold below.

Two things drive the Recheck card. Both are bounded — six steps by default — and read-only: the prompt forbids changing anything, and the same write gate that guards an investigation still applies, so a recheck cannot touch your estate.

The change request's validation plan. Every change request an analysis drafts carries the list L6 checks, fixed when the request is drafted and approved with it: each changed resource with the read that observes it and what it must show, the plan's declared criterion, and every service one hop away on the dependency map in both directions — upstream (it calls the changed service) and downstream (the changed service calls it) — at most 10 items, with the expected blast radius from the impact data. After the fix, validation runs exactly that list: the platform first reads each listed changed resource itself, on the step's own connection; the verify agent is given only the listed items, with no web research, and anything else it reports is dropped; every item ends with a result — pass, fail, not observed — and a changed resource the platform read as failing decides the verdict. A neighbour that fails keeps the change from validating: the verdict reads uncertain and a person follows up — it never makes the change still present or starts a rollback on its own. A standard change from a pre-approved runbook is validated the same way. The incident's L6 layer lists the items (V1 · changed resource, V3 · neighbour, upstream, 1 hop, …) with their results, and the change page shows Validation plan — the only checks L6 runs on its Validation plan card. Requests drafted before the plan existed are validated as before.

Code first, the agent last. Before any agent run, the platform answers what it can at no cost (Guardrails → Validation checks (L6)): a typed cloud fix — a security-group rule, an NSG rule, S3 Block Public Access, the IAM password policy, blob public access, IMDSv2, a Lambda runtime upgrade (the function's configuration and its alias) — is checked by reading the resource it changed again, 30 seconds after the write; a fix whose alerts all cleared at their source after the write is Cleared as soon as the clear arrives when every changed resource on its validation plan reads as expected (an instance the fix started reads running) and no listed neighbour fails — anything less waits for the agent's check on the usual schedule; a check beside the incident's own running analysis waits one interval; a target no read connection can reach gets one $0 cannot verify row; and an Uncertain answer is not paid for again until something changes (after a close: not at all; on an open incident: after 30 minutes, then 1, 2 and 4 hours). For 10 minutes after a fix's start, reboot or restart step an automatic check writes a $0 Not verified row (SKIPPED — no verify run yet: a start or reboot ran less than 10 min ago …) instead of starting the agent: status checks and readiness take minutes after the write. Any other check after a fix is never held. Verify now always runs the agent.

Validation by metric, in rounds. When the workspace has a Post-change validation rule (Guardrails → Change safety) matching the change, its write opens a window — a direct write when it runs, a pull request when it merges — read in rounds: three by default, one every five minutes, the first once the write has settled (Rounds and Seconds apart on the same card; one round of the rule's old minutes reads about the span of the single window from before, but an incident's fix now validates only once its alerts have also cleared at their source). Each round reads every rule over its own span, and the changed resources' compiled state reads too; the change page shows Post-change validation with each round. While the rounds run the card shows a $0 Not verified row VALIDATING — round 2 of 3 (1 clean so far) … and no agent verify runs; the incident's L6 reads deterministic. A rule fails a round when its metric is above the threshold in every minute of it (every period for CloudWatch, read three minutes after the round closes); it passes when the latest reading is back at or below. Every round failed is Still present (the change fails, the incident is escalated — or stays with whoever escalated it — and with Reopen a failed fix one severity level higher on its severity goes up one level and its priority is recalculated; with Roll back a failed cloud fix automatically on, a captured before-state within the allowed tier is put back without waiting for a person); every round clean, with the incident's alerts cleared at their source, is Cleared. Every item of the change request's validation plan (V1, V2, …) has a row in every round, and the L6 list shows the last round's: each changed resource as the rounds' own read of it found it (not observed when no round could read it), the declared criterion as the incident's own alarms read in that round (with no alarm watched, pass when the round was clean), a neighbour as its own signals (not observed when none was read); anything else — a round with no reading, rounds that disagree, a neighbour failing — hands the fix back to the agent verify, which takes a fix waiting for it before the workspace's other open incidents. When only round 1 read nothing and every later round was clean, one more round is read in its place first: the first round opens a few seconds after the write, often before a started instance has published anything. A state read alone (the instance reads running) never cleans a round, and no other signal cleans one while one of the incident's own alarms has not passed: an alarm that read no data fails the round when it treats missing data as breaching and CloudWatch reads it in ALARM (a stopped instance publishes nothing, so its alarm stays in ALARM; data that is only late leaves the round not observed), reads OK when it treats missing data as not breaching (an error count with no errors) without cleaning the round on its own, and leaves the round not observed otherwise. An alarm whose period is longer than a round is shown but never holds one, and the alarm of an alert merged or folded in from another service counts as a neighbour unless the fix changes its resource. A CloudWatch alarm is read as CloudWatch holds it (cloudwatch:DescribeAlarms through the read connection: its metric, dimensions, statistic, period, threshold and missing-data treatment), never as the notification that delivered it describes it; one CloudWatch does not confirm — no metric alarm of that name in the account, another ARN, other dimensions or another service= in its description than the notification's, or a read the connection may not make — only flags a round, like a neighbour. A minute with no reading never counts as above: one burst of errors in a quiet round does not fail a fix. With no matching rule, Watch the signals of a fix that no validation rule matches (Validation checks (L6), on by default) runs the same rounds on what the change request's validation plan names — the incident's open CloudWatch alarms, the changed resources' built-in health rules and their target groups' target health, the neighbours' (which only hold the verdict) — chosen within a minute of the write; a fix whose resources nothing can read (an Azure or GCP fix with no alarm, no inventory) opens no window and is checked as before.

Reachability of the neighbours (AWS, off by default). With the workspace setting aiops_reachability_checks on (API only), such a window also asks AWS Reachability Analyzer whether a path on any TCP port exists between each neighbour that has an EC2 network interface in the estate model and the changed instance — from the neighbour for an upstream one, to it for a downstream one. A path to an instance behind a load balancer's target group is not analysed: its callers reach it through the balancer. The platform creates the path and its analysis through the change's own write connection, both tagged aiops:change with the change's id, and deletes them as soon as the answer is in, also after an error or when the analysis has not finished 60 seconds after its start; every 15 minutes, while the switch is on, a sweep removes any path tagged aiops:change older than an hour, with its analyses. A path found reads pass; no path reads fail with AWS's explanation codes and, like any neighbour that fails, keeps the fix from validating; anything else is not observed. The analyser reads configuration the rounds do not change, so each neighbour is analysed once: later rounds carry that answer (round 1's analysis, carried), and only a not observed one is analysed again. Under Post-change checks — dependents (advisory) the neighbour's row names the level reachability. AWS bills each analysis to your account, about USD 0.10, so one change runs at most aiops_reachability_max_per_change analyses (10 by default) and a neighbour past the cap reads not observed. Reachability alone never opens a window: a fix whose only signals would be paths is checked as before. The write role needs the Reachability Analyzer permissions added in 0.1.146 (re-run the AWS setup for an older role).

The card claims the run in the open: a row appears the moment the agent starts, so you see Verifying now with a 1 agent active badge rather than an unchanged page.

Verdict Meaning Consequence
Verifying now The read-only agent is running. None yet.
Cleared Current evidence says the condition is gone. On an open incident, recorded only — several consecutive Cleared verdicts are your cue to close. On a remediating incident it closes it: an approved fix ran and a read-only agent proved the service normalised. The fix is not credited when every step it executed acted on the resource of an alert merged into the incident and none on the incident's own: the change ends completed with the verdict uncertain and is recorded as cleared by someone else — no last-known-good or memory credit, the change page says why, the incident's Timeline shows Clear not credited to the fix that ran and its remediation history reads not credited.
Still present The condition is demonstrably still there. Recorded as evidence, with an audit row, on an open incident and on a resolved one alike. On a resolved incident it re-opens the incident into escalated; on a remediating one it escalates a fix that did not hold — see below.
Uncertain The agent looked and could not tell. Recorded as-is. Nothing is inferred from ambiguity.
Verify could not run The agent turn never produced a verdict — a gateway out of credit, a provider outage, a restart mid-run. Nothing was measured, so this is not a weak verdict. Not counted as a validation run, and not counted against the post-fix attempt cap. Three in a row park the incident for a growing multiple of the recheck interval (below).
Not verified No agent ran at all: the incident's AI budget refused it, it is waiting out a backoff, its own analysis is running, nothing changed since the last uncertain check, or no connection can reach its target. The row costs $0 and its summary says which. Not counted as a validation run. Grant budget on this page to continue, or press Verify now — the button is never backed off.

Each row carries its timestamp, its verdict summary, and its model and cost when the run reported them. Rows are grouped by the fix they judged, so a verdict is always read against the attempt it followed; earlier attempts fold behind Show validation history for N earlier attempts.

When the executed change failed, or two or more checks on the attempt did not clear, the card adds an Undo the executed change block: a change with an executable rollback links to Review and roll back on its change record, while a recovery action such as a service start has no automatic undo, so the block shows the operator-run command instead. The change record itself shows each step's exact request and the receipt the executor recorded for it under Runbook steps (What ran when the change follows no runbook).

Each group leads with a checklist from its latest verify that recorded one: one line per criterion — what had to hold, what was observed, and a status of pass, fail, unavailable (with the reason) or not observed — with a small badge naming the source of the observation: alert (the alert's own resolved state), receipt (the execution receipt), compiled (the plan's declared checks) or agent (a read-only agent step). A row that came from the alert-resolved path with no agent run says Deterministic check — no agent run, and a run of identical deterministic rows folds behind the newest one as +N identical deterministic checks (every ~M min); expand it to see each row. A row with an agent run behind it carries How the agent checked (N steps · M failed): the steps it took, consecutive same-purpose calls grouped under one purpose line, with a failed step's command and output open by default.

Below the checklist, Post-change checks — dependents (advisory) lists what the verify probed beyond the change's own criteria: the services that depend on the changed one, taken from the service graph's blast radius (each row names the dependent, its distance in hops and the level used). Three levels exist, none of them approval-gated: reachability (a connection probe), HTTP status (an endpoint answers 2xx) and status read (the compiled read through the connector); a full-transaction test — placing an order, for example — is write-shaped, gated like a write and not part of the automatic verify. Every dependent row is advisory: a regressed dependent is shown amber as "regressed (advisory — does not change the verdict)", and the verdict itself is graded on the change's own criteria only — a dependent nobody could observe is shown as unavailable and counted into the manual test plan, and the verdict stays uncertain, never Still present, so a false negative on a dependent can never fail a good change or arm its rollback. A line above the rows says what the verify was checked against; "No dependents in the service graph" means nothing beyond the change's own criteria was checked. When one or more dependents could not be probed, a single callout — N dependents could not be tested automatically — prepare a manual test plan — names them once. A change captured from an external source has no declared criteria; its verification, when the workspace setting aiops_verify_external_changes is on (off by default), is dependents-only and marked Advisory only on the change page, and nothing is ever reverted automatically for it. A Learned checks for <service> block, when present, came from an earlier incident whose corrective actions named a test; the skill it lives in is in Skills.

On a resolved incident, a Still present verdict re-opens the incident: it moves to escalated, resolved_at is cleared and the audit row is written as incident.reopened with the reason "verification found the condition still present after the incident was closed". A Cleared or Uncertain verdict leaves the incident closed and is recorded as incident.recheck. This is the point of verifying a close at all — a close is a claim, and an unverified claim is how a still-broken workload hides behind a green row.

Two invariants that do hold: a recheck never auto-resolves an incident nobody fixed — the one close it performs is the post-fix path, where a human approved the fix, the route ran, and the verdict proved the service normalised — and a claim never outlives its run — a crash, a cancel or a disabled pillar closes the row as Verify could not run rather than leaving a phantom agent on the page.

Rechecks cost tokens. The interval, the batch cap of five per tick and the post-close window are platform settings; on a large open backlog they are the main background spend.

Next steps