Troubleshoot incidents
Incident symptoms, causes and fixes, the limits and gotchas of the incident page, and the permission each action needs.
Troubleshooting · Updated 2026-10-26 · Describes release 0.1.164
Symptoms specific to incidents with their cause and what to do, the limits worth knowing before you rely on a behaviour, and the permission behind every button. The cross-product symptom index is the Troubleshooting chapter.
Troubleshooting
Symptoms specific to this page. The cross-product symptom index is the Troubleshooting chapter.
| Symptom | Cause | What to do |
|---|---|---|
| Review fix or Review pull request is missing | No route matched and a step of the proposal has no supported operation, the incident is resolved/remediating, execution is in computer-assist mode, or your role holds neither incidents.act nor approvals.decide |
Read the current next step. If execution access is missing and you manage connections, the prepared change offers Connect write access (Connect repository for a repository fix), which opens the connection form already set to the write role |
| The next step reads Not applicable — no runnable fix | No route matched, and at least one step of the proposal is not a supported typed operation — for example a bucket-level S3 setting no operation covers yet. A plan runs only as a whole, so one such step leaves nothing to run | The remediation layer lists each write connection and what it still needs (execution consent, a fresh test, a confirmed scope), or says the control has no typed fix. Run the manual steps in the proposal with your own access, then Mark resolved; Re-analyse if the estate has changed since. When the proposal holds no manual step either (only verification commands), the panel says so: fix it with your own access, then Mark resolved |
| A finding reads A prepared fix is available — waiting for a person to start it | The finding is P3/P4: a prepared fix exists, but a P3/P4 never starts a paid analysis on its own | Choose Analyse to start the analysis that drafts the fix (incidents.act), or leave it for a quieter time |
| A Backup or Patch Manager finding reads Commands ready on each resource — a person runs them | The platform holds no write path for the platform's own Backup and Patch Manager reads: each resource on the Resources card lists the commands that close it | Run them in order with your own access; the next daily read (or Refresh reports on Scores) clears the finding once the resource reads compliant. Analyse starts an analysis instead |
| A finding reads Not applicable — no prepared fix yet | No agent has analysed it: the platform has no prepared fix for the finding type and it is P3/P4 (or Automatic analysis of P1/P2 findings is off in Guardrails), automatic compliance analysis is off, paused or over budget, or it is a check of platform settings | Follow the line under the headline — for a platform-settings check, change the named setting; or choose Analyse to start a run (incidents.act; normal cost and approval policies apply) |
| The fix plan still names a resource you suppressed | The plan was drafted before the suppression | Such a plan cannot be released. Wait for the fresh analysis the suppression started, or choose Re-analyse |
| A GitOps remediation fails at the end with a 403 | The repository token lacks Contents: Read and write; repository metadata reporting admin: true is the user's permission, not the token's |
Re-mint the token with write access to repository contents and update the datasource credential |
The pull request merged but the incident is still remediating |
Deployment or recovery has not yet been verified | Check the recorded merge/deployment evidence and run read-only verification: Cleared closes a remediating incident; Still present escalates it |
| An investigation failed or was stopped | The run ended before it could complete its analysis | Read its error and fix the cause, then select Re-analyse on the next-step card, or Re-analyse with all alerts on the Pipeline card. It starts a fresh analysis of all recorded alerts; normal investigation costs apply. Re-analyse is unavailable while another run is active. |
| Export RCA fails with no RCA to export | No investigation on this incident has finished a wrap-up yet | Wait for the run to finish; if it failed, fix the reported cause and select Re-analyse |
| Export RCA, Mark resolved or Mark as noise is unavailable | Your role lacks incidents.act, or the incident is already resolved for Mark resolved and Mark as noise |
Ask an operator or owner; read-only investigation remains available with the appropriate viewing permission |
| Mark resolved returns incident already resolved | Someone closed it first | Refresh; the provenance banner names how it closed |
| Needs you is empty but an agent is paused | The queue may have failed to refresh, a source may be partial, or the record may have changed | Read the queue warning, refresh and open the incident to inspect the current pending approval |
A closed incident came back as escalated |
A Still present verification re-opened it: the condition was still there after the close | Fix the underlying problem, then Mark resolved again; the new close is verified the same way |
| The same problem keeps producing new incidents instead of re-opening the old one | By design — correlation joins any incident that is not resolved | Follow the reopened tag on the incident to the earlier one (the same alert within 30 days; repeated links the same issue on another key — for a cloud alert that names its resource, on that same resource) and its record of whether the earlier fix held; search the list by service for older ones. A reopen window (aiops_reopen_window_h) reopens the last one instead |
| Approve returns approval already decided | Another approver decided first | Refresh; the decided card shows who decided and what ran |
| Edit is not offered, or an edit is rejected as not editable | The tool is an API or MCP call with no command line | Approve or Deny it as proposed; deny with a reason if it is wrong |
| The approval disappeared and the run says approval expired | Nobody decided within the approval TTL, 24 hours by default | The incident escalated; handle it manually, or choose Re-analyse on the incident (incidents.act) to start a fresh run |
| An investigation is stuck at The agent is thinking… | The model gateway is retrying with backoff, or the run is between steps | Wait; the feed advances on its own. Persistent stalls are a platform-side model issue — tell your platform operator |
| An investigation stays queued and never starts | Your tenant runs its agents in a separate runner process and none is currently claiming work | Ask your platform operator to check the runner for your tenant |
| An incident escalated with pillar disabled or unconfigured | That pillar's agent is switched off, so its run could not start | Ask an owner to enable the pillar agent — see the Agents and guardrails chapter — then choose Re-analyse on the incident (incidents.act) |
| The page stops updating | Polling engages only while a run or recheck is live, and never in a hidden tab | Bring the tab to the front, or reload to pick up a run that started elsewhere |
$0.0000 cost on a real run |
Your gateway's model ids are absent from the pricing map and the fallback per-million rates are left at zero | Tokens are still counted accurately; ask your platform operator to set the fallback input and output rates |
| An alert never became an incident | The webhook source is not landing, or the alert was deduped or suppressed | Check the Service correlation page for the fingerprint and its status — see the Dashboard, Operate and alerts chapter |
Limits and gotchas
- A Still present verdict on a resolved incident re-opens it. The incident returns to
escalatedwith its close timestamp cleared, so it reappears in the open work. If the incident had already been escalated by someone else while the verify ran, the verdict is recorded and the escalation left alone. - Needs you counts across history. A partial or failed source is shown explicitly. Use the incident or change record for the current decision and execution state.
- Outbound notifications cover incident open, escalate and resolve (and, on chat, email and webhook targets,
goldenandrollback) — not the approval gate itself. If an approver must be paged the moment an agent stops for a decision, poll the pending-approvals call from your own on-call tooling. - On the shipped default posture you will rarely see an approval at all — mid-diagnosis writes are denied, not paused.
- Writes denied during diagnosis are not listed in the RCA. They appear only as
deniedrows in the tool timeline. - On Incidents, search, filters and sort run on the server over the whole result, 50 rows per page. On Service correlation the status chips query the server too, but the search box and column sorting act only on the 50 rows on screen.
- An incident sits in
awaiting_approvalas a whole, even when only one of several proposed calls is gated, and it stays paused until every pending approval on the run is decided. - Chat, estate x-ray and recheck runs never appear in the investigation panels. Chat and x-ray transcripts are private to their own surfaces by design — their investigation and stream endpoints return not-found — and recheck runs are not attached to the incident, surfacing only in the Recheck card.
- The live stream only exists while a run is live, and five-second polling only runs while something is working and the tab is visible.
- Escalation takes the incident out of the agent loop. Nothing further runs on it until a new analysis is started. The next-step card offers Re-analyse (
incidents.act) when the latest run failed or was stopped (an expired approval, a disabled pillar, Stop this run), when a plan was drafted before a resource was suppressed, when a lapsed suppression reopened a finding, or when the incident was reopened; after a hand escalation of a run that completed there is no button, so a forgotten escalation is a stalled incident. - A resolved incident is not re-opened by a new alert unless the workspace sets a reopen window (
aiops_reopen_window_h, off by default). Otherwise you get a new incident, tagged reopened, repeated or new. $0.0000is a gap in the pricing map, not a free run. Tokens are counted; dollars are not, when your gateway's model ids are missing from the pricing map and the platform's fallback rates are left at zero. Every cost figure on this page and in the executive report is affected.- Artifacts are append-only. The incident's Artifacts card lists all of the incident's own artifacts and its runs' RCA exports; the Artifacts page reads only the 100 newest tenant-wide, and older ones there are reachable by filtering on source over MCP or REST.
- Rechecks cost tokens. The interval, the batch cap of five per tick and the post-close verification window are platform settings; on a large open backlog they are the main background spend.
Permissions at a glance
| Action | Permission | Builtin roles that have it |
|---|---|---|
| See incidents, alerts, investigations, RCAs, artifacts | incidents.view |
owner, operator, approver, viewer, executive |
| Escalate | incidents.view |
all five |
| Review fix, Mark resolved, Mark as noise, Export RCA | incidents.act (Review fix also shows to approvals.decide when the next step is the release) |
owner, operator |
| Release a prepared fix (Approve & run, Approve & create PR) | approvals.decide |
owner, approver, executive |
| Decide an approval (Approve, Deny, Edit) | approvals.decide |
owner, approver, executive |
| Send a suppression on a control's flagged resource (Suppress N selected… / Request suppression for N selected…) or Extend it — a request, not an immediate change | incidents.act or approvals.decide |
owner, operator, approver, executive |
| Approve or reject that request (not your own while another approver exists), and Revoke a live suppression | approvals.decide |
owner, approver, executive |
| Rate an RCA, send a correction | feedback.submit |
owner, operator, approver |
| Read Operate → Needs you | incidents.view; the sidebar entry shows to holders of incidents.act or approvals.decide |
owner, operator, approver, executive |
Unauthorized reads return not-found rather than forbidden, so an incident you cannot see is indistinguishable from one that does not exist. That is deliberate. The full matrix, custom roles and navigation permissions are in the People, access and audit chapter.