Troubleshooting hub

Find the failed step and the evidence needed to repair it.

Troubleshooting · Updated 2026-10-26 · Describes release 0.1.164

This chapter is the page you open when the console does not match your expectations. Every entry names what you see, why the platform behaves that way, exactly what to do, and which chapter carries the full explanation.

Most reports fall into one of three families: something the platform deliberately did (deduplication, a denied write, a hidden feature), something that is not configured yet (no model gateway or schedule), or something genuinely broken (an unreachable endpoint, an expired token). The tables below are ordered so you can tell them apart quickly.

Troubleshooting by section

Each part of the guide keeps its own symptoms next to the procedure they belong to:

Read these two facts first

They explain more support tickets than anything else in this chapter.

Writes. On the shipped default the agent's write is refused mid-diagnosis and returned to you as a remediation plan; your platform operator can switch the fleet to pause mid-run instead. Either way no write reaches your estate without a named human (the one exception, off by default, repeats unchanged a fix a named approver certified: Automatic API fixes for proven runbooks). If you never see an approval card, you are on the refusal posture and that is correct. Full explanation: Guardrails and cost limits → Which posture your deployment runs.

Navigation. The sidebar groups — Dashboard, Operate, Security posture, Insights, Agent quality, the Settings groups and Help — show only the pages your role can use. The API checks authorization again on each request. If access changed recently, refresh the profile and confirm the selected workspace; direct links do not bypass permissions.

Nothing happened after an alert

What you see Why What to do
Your alert source reports 401 invalid webhook credentials Wrong tenant slug, wrong webhook token, no ingest token ever minted, or the tenant is disabled. The four are deliberately indistinguishable Re-copy the URL from the Webhook ingest card on Alert intake & tokens and put the token you hold in place of <webhook-token>: the card never shows a saved token. Mint a new one only when nobody holds it, and update every sender at once. See Webhook reference
Your alert source reports 404 unknown webhook source The path segment is not a supported adapter. Supported: alertmanager, generic, cloudwatch-sns, azure-monitor, grafana, datadog, dynatrace, robusta, servicenow, jira, azure-devops, securityhub, defender Correct the path. Anything without a native adapter posts to /webhook/generic/<slug>
The webhook returned 200 but no alert row appeared The alert was deduplicated. A fingerprint that is already active on an active incident does not create a second row Read the response body: it counts alerts_created, deduped and suppressed (joined for a ticket). A deduped count is the platform working
The alert is on the Service correlation page with status Related — grouped (suppressed) It correlated into an existing incident on a shared service label (or namespace / alertname only while the triage queue is off) — there is no time limit on that join. This is the noise collapse, not a loss Open the incident it joined. Its alert count includes yours
Services failing together stay separate incidents Grouping follows the dependency graph, never a count: with no recorded edge between them they are separate stories. On arrival only edges allowed to fold group them (declared by default), and only while the caller's incident is still open or investigating: a P1 caller does not wait for its dependency, and a dependency alerting after the group window (300 s) finds the caller's analysis started; the grouping agent, on by default, needs a group window to pass, 5 live alerts and budget Record or declare the dependency, then follow How these were grouped on the root and incident.grouped in the audit log; check aiops_storm_agent. Slack, Teams and email digests are a separate notification limit; see grouping outcome checks
The incident opened but no analysis started, and its heading reads Held for triage — unidentified service — no service label The alert named no service; the triage queue is on by default since 0.1.133 (aiops_unmapped_triage) Choose Analyse on the incident. For the alerts that follow, map the source on the Unidentified service — map this source card or add service= at the source. See Unmapped signals
The incident opened, or its alert came back, but no analysis started, and its heading reads Held for triage — grouping window: … The grouping wait: the service depends on another over a dependency allowed to fold, so its analysis waits up to aiops_grouping_wait_s (300 s) in case that dependency fails too and takes the incident in Nothing, or choose Analyse to start it now. The analysis starts by itself when the wait ends; see Alert storm
An incident opened but the investigation shows failed with pillar disabled or unconfigured The pillar agent that alert routed to is switched off. Alerts labeled severity=security, GuardDuty threat findings and alerts from Falco, Trivy, Wazuh or CrowdStrike route to SecOps; other Security Hub and Defender findings go to Compliance (Security posture → Posture); everything else routes to Incident Turn on Enabled for that pillar on Agent behaviour (/portal/agents). The incident was escalated to a human when the run failed
The incident opened and the investigation sits at running with no tool calls at all Either your tenant already has 3 investigations running (the per-tenant concurrency cap) or the deployment runs agents in remote runner processes and no runner is deployed for your tenant Wait for a slot. If nothing moves for minutes and no other run is active, ask your platform operator to check the runner. See Platform operations
An alert with status: resolved did nothing Resolution payloads close alerts, they never open incidents. An incident auto-closes only from open or investigating, never from awaiting_approval or remediating Expected. A human holds those two states

Procedure: confirm an alert actually arrived

Requires incidents.view.

  1. Open Operate → Service correlation.
  2. Find the row by its alertname label or its fingerprint.
  3. Read the Status column: Received, Root alert, Related — grouped or Cleared (API values received, correlated, suppressed, ignored). Cleared rows are hidden until you pick the Cleared or All filter.
  4. If the row is Root alert or Related — grouped, open the linked incident.

You should see either your alert on the Service correlation page with a status that explains its fate, or no matching row on the loaded page. If no row matches, check the HTTP response, workspace, filters and older pages; an exact active repeat is deduplicated without a new row. Absence alone does not identify a credential failure.

The analysis reads generic and cost shows $0.0000

What you see Why What to do
The investigation header reads mock, the token count is a handful, cost is $0.0000, and the root cause literally begins Mock root cause for: No model gateway is configured on this deployment. The platform falls back to a deterministic scripted responder so the whole click path stays demonstrable without a key Ask your platform operator to configure the gateway, or the workspace owner to review its model tiers and Bring your own LLM endpoint on AI models. A private agent also needs a reachable gateway or explicitly shared workspace gateway. See Models, tiers and routing
The Test Lab Replay tab labels its runs mock runner Same cause. With no real model the platform uses a ground-truth echo runner and says so, rather than reporting a meaningless pass rate Scores become meaningful once a real model is configured
The header names a real model and a realistic token count, but cost still reads $0.0000 Your gateway's model identifiers are absent from the pricing map, and the platform's fallback per-million rates are left at zero Token counts remain accurate and are the honest cost metric. Ask your platform operator to set the fallback input and output rates. See The model gateway
You set a cost cap on Guardrails & cost and a long run blew past it A cost cap cannot fire on a cost that is always zero. With no pricing, the run is bounded only by the step cap Fix the pricing first, then the cap works. The same applies to the estate x-ray budget. See Guardrails and cost limits → The cost cap

A run failed with a model provider error

The run shows failed, its header shows 0 tokens · $0.0000 after either the model id or no model response, and the run text carries the provider's refusal. no model response is not the scripted mock: it means no model answered this run. The platform retries a failed run a few times before it marks it failed, so the error appears after the last attempt.

What you see Why What to do
engine run failed: PermissionDeniedError: … is not available for this account (on an estate x-ray scan: PermissionDeniedError: … is not available for this account) The model provider refuses this model for the account the platform calls it with. On Amazon Bedrock, model access is not active in that AWS account: the Anthropic use-case form has not been submitted, or the AWS Marketplace agreement for the model has not been created. Bedrock creates that agreement on the first invocation of the model, and only when that invocation comes from a principal allowed to subscribe to Marketplace products — the platform's own runtime role normally is not Platform operator: in the AWS account and region the platform uses, submit the Anthropic use-case form if Bedrock asks for it, then invoke the model once from the Bedrock console Playground (or with any principal allowed to subscribe). Then choose Re-analyse on the incident, or launch the scan again. Nothing in the workspace needs to change. On another gateway, check that the key the platform uses is allowed to call this model
engine run failed: AuthenticationError: … The gateway or provider rejected the platform's credential: the key is missing, expired, revoked, or belongs to a different account Platform operator: check the key configured for the model gateway. If the workspace uses its own gateway (Bring your own LLM endpoint on AI models), the workspace owner checks that key instead
engine run failed: NotFoundError: …, or a BadRequestError that says the model does not exist The model name is not one the gateway serves: a typo, a missing provider prefix such as bedrock/, or a model not offered in that region Correct the model name where it is set: the platform model setting, or the tier ids on AI models. The ids must be ones that gateway serves
No LLM configuration resolved for agent job: … Nothing names a model: no platform model, no tier ids on AI models, no workspace gateway. An agent that talks to the platform over its API never falls back to the scripted demo model, so every run fails Platform operator: set the platform model. Or the workspace owner sets the tier ids on AI models. Then choose Re-analyse on the incident

An agent seems stuck

A permitted operator can use Stop this run on an active incident investigation. It requests cancellation at a checkpoint; do not confuse that request with a stopped result. Discovery also has a separate Cancel action.

What you see Why What to do
The tool feed stops advancing and the page says the agent is thinking Normal between steps, or the model gateway is retrying with backoff after a failed call Wait. The feed advances on its own. A stall lasting minutes is a gateway problem, not a tenant one
A run stays running with no progress and your tenant has other runs in flight The per-tenant cap is 3 concurrently running investigations. The fourth is queued and still displays as running Wait for a slot
A run stays running across a platform restart Interrupted runs are resumed from the transcript committed at every step, not restarted from scratch. Chat, benchmark and agent-API runs are not resumable and are marked failed instead Nothing to do. If retries are exhausted, the run fails and its incident is escalated
A run stays active without useful progress A tool or gateway may be stalled Inspect its current step; an operator with incidents.act can request Stop this run. Keep an appropriate cost ceiling under Guardrails & cost
An estate x-ray scan sits at running for minutes with no new steps The current agent, tool or model step may be waiting or retrying; no progress alone does not identify the cause Inspect its ledger/activity and private-agent readiness. Use Cancel if needed; cancellation is polled at step boundaries and does not interrupt an already running external call

A verification is running after closure

A resolved incident can still have a bounded read-only verification in progress. Read its Recheck verdict and timeline. Still present can reopen the incident into escalated; a closed status is not proof that the condition cleared. The Dashboard shows recorded work and readiness rather than a fleet-wide "agents active" tile.

Recheck and post-close verification.

A nav item is visible but the page is empty or returns 403

What you see Why What to do
An incident, alert or artifact page returns 403 requires permission 'incidents.view' Your selected role no longer has incident read access, or you followed a saved link without it Confirm the workspace, refresh the profile and ask an owner for incidents.view if needed
Guardrails & cost loads but Save guardrails returns 403 Reading guardrails needs only membership. Saving needs members.manage Ask an owner to change it. The floors themselves are code and have no setting at all
On Agent behaviour you can edit the model, schedule, prompt and enablement, but Extra approval gates (tool name patterns, one per line) is read-only That field defines the human gate, so editing it needs pillars.manage, which is owner-only among the builtins. Everything else on that card needs incidents.act Ask an owner for the gate change
A settings or Test Lab destination is missing The selected workspace or role differs from the one you expected, or the profile is stale Agent behaviour appears with incidents.act or pillars.manage; Test Lab with benchmark.view or pillars.manage; Guardrails & cost with pillars.manage or members.manage. Refresh after role changes
An incident, datasource or tenant you know exists returns not found Unauthorized reads return not-found so existence stays hidden. Platform staff have no bypass Confirm your membership and the identifier. Staff need an audited support-access grant. See Support access
Your role changed but navigation has not Your session profile is cached for 60 seconds Reload the page

Permission keys are listed in full in Permissions. The exact key you lack is always named in the 403 message.

Estate x-ray or Lakehouse is unavailable

Features are included for all enabled tenants/workspaces. Check that you selected the intended workspace, that it is enabled and that your role has the required read or management permission. Refresh the page after a role change.

A reachable connection, configured model or collection specification is still required for its operation. If an older client says a grant or terms acceptance is required, update/reload it and report the client/server versions; there is no grant or acceptance action to complete.

A datasource tests OK but the agent still cannot read it

Data sources: each connection's type, role (read or remediation only), state and last test, with its actions on the row (Test, Edit and Disable first; Delete only on a disabled connection) and the whole row on screen; Show disabled connections (1) beside the category filter lists the hidden disabled connection; the banner above names which connection runs which job for AWS account 123456789012.
Data sources: each connection's type, role (read or remediation only), state and last test, with its actions on the row (Test, Edit and Disable first; Delete only on a disabled connection) and the whole row on screen; Show disabled connections (1) beside the category filter lists the hidden disabled connection; the banner above names which connection runs which job for AWS account 123456789012.

Test proves only the read or health operation named in its result. A credential that passes can still lack access to the data you need or permission to execute a fix. An execution-only credential without approved-plan consent is saved without an automatic probe. For AWS/Azure with consent enabled, Save and check access performs a safe identity/read test. Then review the actual plan on the incident or Change; no remediation route is required. Private-agent checks apply only to investigation-enabled connections.

What you see Why What to do
Test is green, the agent never uses the connector The datasource is disabled, or its Role is set to write. A write-role credential is remediation-only and is invisible to every agent by design Enable it, or connect a separate read-role datasource. Full explanation: Read and write roles
Test is green, the connector works for one pillar and not another The datasource carries a pillar allowlist, or the connector type is scoped to certain pillars by the platform Clear or widen the allowlist on the datasource
No tool for the connector appears anywhere An API connector was saved without a base URL, so no HTTP tool is registered for it Edit the datasource and supply the URL. Cloud accounts are the exception: they carry credentials instead of a URL and get their own signed tools
The tool timeline shows the call with status denied The call was classified as a write. Reads are an allow-list on top of default-deny, so an endpoint the catalog does not recognize is gated even when it is harmless If it truly is a read, tell your platform operator: the platform records these as unclassified-call audit rows precisely so the catalog can be extended
A suggested or connected endpoint is reported as needs_endpoint The URL only resolves inside a specific network: cluster DNS, a single-label host name, loopback, or a private address range. The platform may not share that network Expose a read-only route, or replace the URL with one the platform can reach
A suggestion is reported as needs_credentials The configuration is valid but required secret fields are empty Fill the named fields. They are write-only and are never echoed back
web_fetch calls are denied Web fetch runs freely only for domains on your web-fetch allowlist. Any other URL is gated, and inside an estate x-ray a gated call is denied outright Add the domain to the allowlist on Guardrails & cost
A tool reports the SSRF guard blocked a private address The guard rejects non-public addresses for arbitrary fetches. Your own connector hosts are exempt automatically Add the host to the tenant's SSRF allow-hosts on Guardrails & cost, or have an owner turn the guard off for your tenant

Scanner, keyless AWS and posture-history messages

What you see Next step
Scanner sync says Security Hub is not enabled, rate limited, partial, or Still running on the server Read the connection's result, permissions, coverage and paging/pacing guidance in Pulling findings from Security Hub and Defender; a client timeout does not establish that the server stopped
AWS refuses a keyless connection's Test Use the full refusal's setup/trust/name guidance in Manage connections and AWS prerequisites; do not treat it as a static-key failure
Posture trend says daily snapshots start at the next UTC midnight, or draws faded periods Follow the saved-versus-reconstructed history explanation in Compliance findings; missing historical snapshots are not zero findings

Procedure: find out what a credential can actually do

Requires datasources.manage.

  1. Open Data sources and select the connector.
  2. For a read connection, run Test and review the reported operation. A write credential does not need to pass this read check before you can inspect its grants.
  3. Run Audit perms on the same connector's row.
  4. Read the recommendation.

You should see either a real introspected answer — the identity, the reads allowed and any destructive writes granted — for AWS, GitHub and Kubernetes connectors, or an honest statement that this connector type exposes no permission API, together with the platform's own read and write classification for it.

Every write pauses for approval

That is the floor, not a bug, and it cannot be switched off from the portal.

The write gate is an allow-list of reads layered on default-deny: a call runs freely only if the platform classifies it as a read. Everything else is gated. Per-pillar settings can only add gates, never remove them. Remediation routes run only after a human decision, and only with a write-role credential the agent itself cannot see.

What you see Why What to do
An approval card you did not expect Your deployment runs the pause posture rather than the refusal posture Decide it on the incident page, next to the tool trace
An incident sits in awaiting_approval although only one of several calls was gated The investigation pauses as a whole and stays paused until every pending approval on that run is decided Decide all of them
Edit is not offered on an approval The proposed action is an API or MCP call with no command line to edit Approve or deny it as proposed
The approval vanished and the run reads approval expired Nobody decided within the approval time-to-live, 24 hours by default. The approval was denied by timeout, the run failed, and the incident was escalated Handle the incident manually. There is no way to revive the expired run
Nobody noticed the approval in time Outbound notifications fire on incident open, escalate and resolve (chat, email and webhook targets can also take golden and rollback) — not on the approval gate. The Needs you page polls every 15 seconds while someone has it open Turn on notifications under Settings · Account → Notifications & on-call for the incident events, and wire the pending-approvals listing from the MCP server or REST API into your paging for the gate itself. See REST API → What REST does not have
An approver's queue is empty while an agent is paused The Needs you queue is served and counted server-side (GET /{tenant}/needs-you); an empty queue next to a paused agent means a source failed to load (a red banner names it) or the run holds no pending approval row any more (stale_gate — the row says so) Retry from the banner, or open the incident directly. The incident page is authoritative: for stale_gate its next step reads Stop this run, then re-analyse or resolve — choose Stop this run, then Re-analyse or Mark resolved. See Incidents

Integrations: MCP, tokens and webhooks

What you see Why What to do
Your MCP client cannot connect at all Wrong endpoint. The tenant server is at /mcp/aiops on the API host, and works with or without a trailing slash. The base /mcp prefix alone returns not-found Copy the workspace endpoint from Settings · Data in → Connect your agent → Use your own agent
Your MCP client hangs while connecting and never lists tools The client is set to the legacy HTTP+SSE transport (sse). It opens a GET stream and waits for an endpoint event this server never sends. The server speaks Streamable HTTP only Set the client's transport to http (Streamable HTTP), as the generated configuration does
The client connects and every call returns unauthorized No bearer token, a malformed header, or a revoked, expired or unknown token Send Authorization: Bearer with a personal access token or your SSO access token. Tokens are shown once at creation; your API tokens list on Alert intake & tokens marks a revoked or expired token revoked or expired
A tool returns token scope is insufficient The token's own ceiling, independent of your role Mint a write-scope token. Scope cannot be changed after creation
A tool returns 400 Pass tenant with a workspace ID or slug An unbound client has ambiguous workspace membership Use the generated /mcp/aiops?workspace=<slug> URL, or pass tenant with the workspace ID or slug in the call
Tools you cannot use are still listed The catalog is not filtered by permission. The gate is the per-call response Do not treat a visible tool as an entitlement
Every alert source broke at once after you minted a new webhook token Minting replaces the previous token immediately, with no overlap window Update every source in the same maintenance window

Full request and response detail, including every error code, is in the Webhook reference.

Notifications arrive as a digest

What you see Why What to do
I got one digest instead of every message: one message, then [AIOps] N incident updates in the last 2 min Burst grouping. A Slack, Teams or email target gets at most one incident message per interval — 120 seconds unless Message interval (seconds) on Settings · Account → Notifications & on-call says otherwise. Later events are held and sent together when the interval ends: P1 first, escalations first within each priority, at most 20 lines plus a count Expected. Open Operate → Incidents for the full list. For a message per event, set Message interval (seconds) to 0 (see REST API → What REST does not have), or add a webhook target: webhooks are never grouped
A message arrived up to two minutes after its incident changed It was held behind an earlier message to the same target. While a digest is pending, newer events join it, so nothing overtakes an older held event Expected during a burst
A digest never arrived Filter the Audit log on incident.notified and find the row with target digest. dropped: true: the target was removed, or its URL or recipients changed, before the interval ended. ok: false: the delivery failed. No such row: the control plane restarted while the digest was pending — held events are kept in memory only Fix or re-add the target. Every held event is still in the log with held: true, so you can see what never went out. See Audit log → Notification setup checks

The portal shows "Couldn't reach the API"

This screen means the console could not load your session and tenant list after several retries with backoff. It is deliberately distinct from No tenant membership, which means the API answered and you genuinely have no tenant.

  1. Press Retry.
  2. If it fails again, wait about a minute and reload.

You should see your normal console. During a platform upgrade this screen is expected: the control plane runs as a single replica, so a rolling deploy is a brief full API outage.

Work in flight is not lost. Agent runs are stored as durable rows and are resumed from their committed transcripts when the platform comes back. On deployments that run agents in separate runner processes, jobs continue right through a control-plane restart.

Scheduled agents never fire

What you see Why What to do
An agent card reads Event-driven Its Schedule (cron) field is empty. No schedule ships enabled by default Enter a cron expression and save. Requires incidents.act
The sweep ran hours later than you intended Cron expressions are evaluated in UTC, always. There is no per-tenant timezone Convert your intended local time to UTC
The schedule did not run while the platform was down, and never caught up There is no catch-up. A missed window is skipped, never replayed, and a newly configured schedule fires from the moment you saved it rather than from a historical base Trigger the run manually if you need that window covered
The schedule was due but nothing fired A previous scheduled run for that pillar is still running or paused, so the fire is skipped. At most one run per agent per tick Finish or decide the previous run
A saved cron expression silently does nothing The scheduler cannot read the stored expression, so it logs and skips it every tick. Saving refuses such an expression, so this one was saved before that check existed Re-enter it in standard five-field form, for example 0 2 * * *, and save; an expression the scheduler cannot read is refused with schedule_cron '<expression>' is not a valid cron expression
Schedules stopped firing entirely across the tenant Schedules fire from one control-plane process — the elected leader when the platform runs several replicas — so a platform-side problem stops them for every tenant at once Ask your platform operator. See Platform operations

What to collect before you escalate

The checklist to gather before you ask for help — ids, the run header, failed tool calls, the connection test and the audit entries — has its own page: What to collect before you escalate.

Limits worth knowing before you report them

These are current product behavior, not defects.

Next steps