Platform incident response
This chapter handles failures of the platform itself. Page the customer platform on-call through an independent monitoring route; never feed these alerts into the platform's own ingest.
How-to · Updated 2026-10-15 · Describes release 0.1.164
Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).
This chapter handles failures of the platform itself. Page the customer platform on-call through an independent monitoring route; never feed these alerts into the platform's own ingest.
Triage and contain
- Record UTC onset, affected hostname, health version/result, workspace slug and incident or investigation id. Preserve the last deployment receipt and relevant monitoring evidence without credentials or raw secret output.
- Determine scope: portal/API, intake, one runner, all runners, model calls or an estate endpoint. Ops → Overview → Agent runners distinguishes no ready runner, expired leases and exhausted retries.
- If agent activity must stop, the workspace owner uses Pause all agents on Guardrails & cost. Intake still records alerts; blocked starts escalate to humans. For one active investigation use Stop this run; check for the terminal result, since the button requests a checkpoint cancellation.
- Apply the corresponding check and fix below. Route to the team owning that component; preserve evidence before a restart or rollback.
Symptom → check → fix
| Symptom | Check | Fix and owner |
|---|---|---|
| Queued investigations, runner not ready or Pending | AiopsRunnerNotReady, Ops runner panel and scheduling event; CPU/memory requests, node capacity and any NodePool CPU ceiling |
Cloud/platform team frees approved capacity or adjusts its capacity limit. Keep the /data zone constraint in mind for the control plane. Increasing spend limits cannot schedule a pod |
| Claimed run stops renewing its lease | AiopsRunnerSilent, newest heartbeat, runner Deployment readiness |
Platform team checks the runner failure. Durable eligible work resumes from its committed transcript after the 180 s lease, at most three attempts. Exhausted work needs a human and a new run; paused approvals are separate |
| API/intake errors or database connection exhaustion | AiopsDbPoolExhausted, RDS connections alarm, pool size plus overflow and active concurrency |
Cloud/platform team checks RDS reachability, capacity and leaked/long-held connections. Tune AIOPS_DB_POOL_SIZE/AIOPS_DB_MAX_OVERFLOW against the whole database limit, or reduce AIOPS_MAX_CONCURRENT_INVESTIGATIONS; more pools can worsen RDS exhaustion |
| Cloudflare error, redirect loop or ALB TLS failure | DNS target, proxy mode, ACM status and hostname covered by the certificate; distinguish CDN-to-origin TLS from client-to-edge TLS | DNS/cloud team uses the customer runbook's DNS-only CNAME and valid ISSUED ACM certificate. Flexible proxy TLS loops on the ALB redirect. Imported certificates need operator renewal; keep ACM validation CNAMEs |
| New runs refuse to start with a licence reason | Ops → Platform licence state, expiry/grace, public key and token pairing | Platform/licence owner follows Licence. Valid renewal and rollout restore starts. Webhooks, login, history and exports remain available; rollback and ticket paths remain available |
| Model gateway unavailable, credential rejected or model not found | Investigation error and model id; gateway health/authentication, endpoint and region, account entitlement | Gateway owner restores service or credential; platform operator corrects the configured id. Bedrock account administrators complete agreements/use-case onboarding; runtime roles cannot do it. Re-analyse only after the cause is fixed |
| One active investigation has no useful progress | Current step, tool response, gateway retries and booked cost | Authorized operator requests Stop this run and confirms cancellation. Discovery uses its own Cancel action. Do not delete durable database rows to clear a queue |
| Budget trip or kill-switch block | AiopsBudgetTrip, Pause all agents, daily/monthly/per-incident limits and model pricing |
Workspace owner reviews the reason and spend, then deliberately resumes or adjusts a justified limit. Alerts already recorded still need human handling |
| No scheduler leader or audit sealer stalled | AiopsLeaderLost, AiopsAuditSealBacklog, scheduler tick timestamp and database advisory-lock connection |
Platform team restores control-plane/database health; verify chains after sealing resumes. Preserve an external archive anchor if verification reports tampering |
| All HTTP traffic fails after a release | Deployment result, ALB health and latest image/chart/key pair | Release team follows Rollback. Avoid Helm revisions predating a key rotation; data restore is a separate controlled recovery |
| Control plane restarts during long estate harvests | Running image and version, restart reason, timestamps | 0.1.115 ran out of memory on a long-harvested estate; install 0.1.116 or later through the normal upgrade path. Do not replay a write whose outcome is unknown |
| A run was refused by the model route policy | The Model denied by route policy audit row with its alias, backing model and region; the workspace's residency policy; agent version | Select an allowed in-region route or correct the documented policy. Do not switch the policy off to make a run pass |
| Bedrock refuses the prompt-cache markers | Exact error, selected alias, deployed version | 0.1.120 retries without the markers and remembers that route's refusal for the process lifetime. Check the gateway entry; a re-run is a new spending action |
| Price sync fails or an alias stays unpriced | The fetch result on Model price map, egress to pricing.us-east-1.amazonaws.com and models.dev, the Needs attention reason |
Repair the destination or the mapping, or record a manual price with its source note. Keep Not priced yet and estimate labels until a supported price exists |
| Built-in detection does not evaluate | The workspace's own detection setting; the platform switch Built-in detection may run for workspaces that turned it on in the Ops console; the connection's job | Identify which gate is off before changing it. The platform switch is a freeze: off, nothing is evaluated and nothing is resolved, so open detector alerts stay open |
| The retention sweep did not complete | retention_last_sweep with its error, the preview and cutoff |
Preserve the failed result; it is retried a day later. Do not delete rows by hand. See Security operations |
| A private-network agent ignores a newer setting | Control-plane and deployed agent versions, the runner's last heartbeat, one scoped read | Roll the agent to the matching release, then verify a scoped datasource read. A heartbeat alone is insufficient |
| Senders get 401 after a webhook token was minted | Mint time, each sender's configuration revision, delivery status | Update every sender with the new token through the approved secret channel; there is no overlap period. Keep the value out of logs and recordings; do not mint again |
The seven starter expressions and their corrected first-trip/zero-pool conditions are in Monitoring. Thresholds are starting points, not contracted response targets.
Read-only evidence collection
The following pod commands are normalized from the tested monitoring verification and its fact sheet; use the installed namespaces. Run them from the approved EKS admin context. Logs and pod descriptions can contain connection information: review locally and sanitize before sharing.
kubectl -n aiops-agents describe pod
That scheduling check is quoted by the monitoring guide's AiopsRunnerNotReady rule. Prefer the Ops panel and the monitoring system for routine evidence. Do not export pod environment, Secret manifests or a full private log into a support ticket.
Escalation and closure
Customer operator → customer platform/cloud on-call → vendor support → release or engineering lead. Use the named contact and response arrangement in Support and escalation; response and restoration targets must be agreed, not inferred from this runbook.
Send onset UTC, scope, health version, latest deployment id, sanitized error, workspace and run ids, monitoring changes and actions already taken. Grant time-bounded support access only when workspace evidence is needed, and revoke it when the investigation closes.
Close after affected paths pass release verification, queued work is accounted for, intake errors clear and audit sealing advances. Record cause, actions, timing and a prevention owner. A pod becoming Ready alone is not a complete recovery check.
Sources: Deployment runbook §1.3, §1.5, §4 and §12, Availability, monitoring guide, verified monitoring record, and in-product troubleshooting.