Operator runbook: changes, runbooks, safety and sources

Changes and their classes, runbooks, the safety controls and spending limits, and connecting alert sources, cloud accounts and scanners.

How-to · Updated 2026-11-02 · Describes release 0.1.184

5. Changes

Every fix the platform makes is a change request (CHG-…). It carries a class, a tier, a scope, the approvals it needs, the before-state, a validation plan and a rollback plan. This section is how you release, check and undo one.

Where: Changes › Changes (/portal/changes). One change opens at /portal/changes/<id>. The incident or finding that raised it links to it from its L5 Remediation step.

Who does what (built-in roles, release 0.1.184):

The operator requests; the approver decides and releases. An approver cannot run Verify restoration only (no incidents.act); ask an operator.

5.1 Read a change before you act

When: before every approval, release or rollback. Who: anyone with incidents.view.

Steps

1. Open the change from Changes › Changes, from Needs you, or from the incident's L5 Remediation step.
2. Read the header card: the title, the status chip, the class badge (Standard, Normal or Emergency) and the tier.
3. Read the stage bar: Request → Classify → Approve → Window → Run → Verify. It shows the approval count (for example 1 of 2) and whether the window is open.
4. Read the panel on the right. While the change is moving it is titled What you need to do. Once it has ended it is titled Result. The filled button is the next step for you. Other actions sit under More actions.
5. Open the folded sections you need:
   • Class, tier and scope: the class, why it has that class (Derived: … when a person raised it), the tier with its approver count, the resources in scope, and Window that applies: <service> → <window>.
   • Plan, Runbook steps, What will run (becomes What ran after release).
   • Validation plan: what decides that the fix worked.
   • Rollback plan: the before-state per step (Pre-state per step) and how the change is undone.
   • Before / after state: the reads of each written resource.
   • Timeline: every event in order.
   • Change request document: see 5.5.

How to check: you can say the class, the tier, how many approvers it needs, which window applies, and how it will be undone.

If not: a section reads Not provided yet when there is no data yet (no plan, no harvest, no approval). Re-analyse the incident, or wait for the change management agent's draft (5.5).

5.2 The three change classes

The class decides who approves and whether the maintenance window holds the change. The platform sets it in code; nobody types it.

Rules:

• A class only rises. A person may raise Normal to Emergency. Nobody lowers a class.
• A class never lowers the tier. A standard change on a production target keeps an L2 floor if the target needs it.
• If the runbook behind a Standard change is demoted, disabled, deleted or edited before the change is released, the change becomes Normal — runbook withdrawn (Emergency at P1). Its earlier approvals no longer count.
• The workspace policy is on Settings › Guardrails & cost › card 1 · Approvals & change classes › fold Change class policy: Derived (the rules above) or Standard by default (every request starts Standard). Leave it on Derived unless your change process says otherwise.

5.3 Raise a change's class (reclassify)

When: a Normal change is more urgent than the platform judged, for example a customer-facing outage. Who: incidents.act (owner, operator). Only before anything is dispatched (status planned, pending approval or approved).

Steps

1. Open the change.
2. Open More actions › Reclassify.
3. In Class, pick the higher class.
4. In Why, write the reason (minimum length is shown under the field). It is recorded on the change and in the audit log.
5. Select Save.

How to check: the notice reads Reclassified to emergency. Tier L3, 2 approvals required, previous approvals cleared — review and approve again. The class cell reads Emergency — set by a person.

If not: no Reclassify item means there is nothing higher to go to. The class cell explains why, for example A certified runbook makes this Standard. To make it Normal, demote or disable the runbook. or Emergency is the highest class; a class is never lowered.

API: POST /api/aiops/{tid}/changes/{id}/class with {"change_class": "emergency", "note": "…"}.

5.4 Maintenance windows

A maintenance window is the time a Normal change may run. With no window declared, releases are always allowed.

When: at onboarding, and whenever your change calendar changes. Who: members.manage (owner). Everyone can read the windows.

Steps

1. Go to Settings › Guardrails & cost. Card 2 · Maintenance windows — When Normal changes may run shows a month calendar of the windows (legend: Window, Bypassed, Ran).
2. Open the fold Edit windows and patch schedules.
3. In the Maintenance windows list, the first row is the Workspace default. Each further row is one service:
   • Service: a name or pattern, for example payments*.
   • Days: for example sat,sun.
   • Weeks of the month (1–5, blank = every week): for example 2 for the second such weekday.
   • Hours: for example 22:00-02:00. Hours after midnight belong to the day the window starts.
   • Time zone: blank means UTC, or an IANA name such as Asia/Singapore.
4. Use Add service window for another service row and Remove to delete one.
5. Select Save windows.

The first service row that matches wins. A service with no matching row uses the workspace default.

How to check:

• The notice reads Maintenance windows saved.
• The calendar shows the new window days.
• Open a waiting change for that service: Window that applies: names the window, and adds — closed now while it is closed.

If not:

• <pattern>: give days, hours or both. means a row is empty. Fill it or remove it.
• An unknown time zone is refused when you save (400).
• An empty or malformed window can leave releases open. Always read back the saved row and check Window that applies: on a real change.

API: PATCH /api/aiops/{tid}/guardrails with, for example, {"change_window": {"days": ["sat"], "hours": "22:00-06:00"}, "change_windows": {"payments*": {"days": ["sun"], "hours": "22:00-02:00", "tz": "Asia/Singapore"}}}.

Patch schedules (the Patch schedules card under the windows) plan a patch run before a window opens. See Change windows, emergency changes and rollback.

A freeze window (change_freeze) is different: it does not block platform releases. It flags changes made outside the platform during that time as unauthorized. See Unauthorized changes.

5.5 The change request document

Every change carries a document in your change request template: version control, requesting organization, then sections 1 to 9 (objective, requirements, impact, risk, environment, scope of work, rollback plan, timeline, running cost, closure, customer acceptance).

The platform fills class, tier, approvals, scope, commands, validation and rollback from its records. The change management agent drafts the narrative parts (objectives, assumptions, impact in plain words). That agent is on by default (Let the change management agent draft change requests, Guardrails › Change safety › fold Change management agent). It costs one short agent run per change.

When: before you approve (the approver signs this document), and after completion for your CAB records or ticket. Who: read and download: incidents.view. Ask for a new draft: incidents.act.

Steps

1. Open the change. Under the two columns, find the folded Change request document. Its summary shows the title, the reference, the submitted date and the number of sections.
2. Open it. Each section carries a badge: Change management agent, Platform records or Agent draft + platform records.
3. Select Download Markdown or Download Word. A completed change also offers Download change request (.md) in the Result panel.
4. If the incident was re-analysed, or you declared the change yourself, select Redraft the document text. The run starts within a minute.

How to check: a file named change-request-CHG-<id>.md or .docx. The line under the buttons says the narrative was drafted.

If not:

• Approve answers the change request is still being drafted: wait. Approval opens again when the draft lands, fails or times out. An emergency change does not wait for the draft.
• the change management agent is off for this workspace (409): the switch is off on Guardrails.
• The drafting line says the change pillar is disabled: an owner turns it on under Settings › Agent behaviour.

API: GET /api/aiops/{tid}/changes/{id}/document?format=markdown (or docx); redraft POST …/changes/{id}/document/draft. Section-by-section sources: The change request document.

5.6 Approve and release a Normal change

When: a Normal change is in Needs you or its incident shows waiting for approval. Who: approvals.decide (owner, approver, executive). L2 needs one approver.

Steps

1. Open the change (or the incident's prepared change in its What you need to do panel).
2. Read the change as in 5.1 and the change request document as in 5.5.
3. Optional: type a condition or comment in Approval note (optional) (up to 1,000 characters). It is kept with your vote, on the incident's L5 line and in the document.
4. Select the filled button:
   • on the incident or finding, window open: Approve & run when your vote completes the quorum (otherwise Approve change);
   • on the change page, window open: Approve the exact action. Its dialog confirms with Record exact approval. Then release it with Review execution › Apply reviewed action (a pull-request change: Create pull request);
   • window closed (either page): Approve — runs in the next window (<time>). One click records your vote and, once the quorum is complete, schedules the release for the window.
5. In each dialog, check the exact operation and the remediation connection before you confirm. For a compliance change the release dialog also offers When to run: Now, At a time (UTC) or the next window; it then confirms with Approve and schedule.

How to check:

• Open window: the change moves to running, then completed. The Run step on the stage bar shows the time it ran.
• Closed window: the header reads Approved · scheduled to run — … and the panel reads Scheduled to run <date> UTC (maintenance window …) with Reschedule and Cancel schedule. At the window time the platform releases it as you, after it checks again that the action, the window and the approvals have not changed. It never runs outside the window.

If not:

• Not run at its scheduled time with a reason: the action, the window, an approver or your role changed. The change stays approved for a person. Read the reason, review it again, and release or schedule it again.
• To stop a scheduled change, select Cancel schedule. The approvals stay.
• To refuse a change, open More actions › Reject, write a Reason, confirm Reject. The plan is not offered again. A re-analysis drafts a new one.

One-time window: if one Normal change must run outside its window, an operator selects Request a one-time window on the change (start and end in UTC, at most 24 hours apart, and a reason). A different approver selects Approve one-time window or Reject (it appears on Needs you as Window exception, chip One-time window). Approving the window is not a release vote. The change still needs its approvals.

5.7 Release an emergency change

When: an Emergency change (P1 or critical, no certified runbook) waits for approval. Who: two different people with approvals.decide. The second vote releases it.

Steps

1. First approver: open the change or the finding/incident. In the prepared change, select Approve change. Nothing runs yet. The page reads Release parked — approved by 1 of 2, needs a second, distinct approver.
2. Second approver (a different person): open the incident or finding, open L5 Remediation, find the prepared change and select Approve & run. Or open the change and use Approve the exact action.
3. The second vote completes the quorum. If the window is closed, it runs anyway and the window step reads bypassed — emergency.

How to check: open the change. Its status reads completed. Open Timeline: it shows Change window bypassed — emergency and Execution receipt. Open Validation plan for the verdict.

If not: the same person cannot cast both votes. If only one approver exists in the workspace, add a second approver (section 14) before an emergency happens.

5.8 Close the post-implementation review

Only an Emergency change owes a post-implementation review (PIR). By default it is scheduled when the emergency change is submitted (Schedule the post-implementation review when an emergency change is submitted, on), and it is due 72 hours later.

When: the change shows Post-implementation review owed — due in X (red overdue by X when late), or Needs you lists it with the chip Review. Who: approvals.decide.

Steps

1. Open the change. The Post-implementation review section is open while the review is open (status, due date, owner).
2. Select Close post-implementation review (the filled button in the Result panel).
3. In the dialog Close the review of CHG-…, write the outcome in Review note (at least 10 characters).
4. Select Close review.

How to check: the review reads closed; the Timeline shows review closed; it leaves Needs you.

If not: the review cannot be closed while the change's outcome is still pending. Record the real outcome first (verify, or reconcile).

5.9 Before-state

Before every direct cloud write the platform reads the resource and stores its current state with a checksum. This is the source for the rollback.

• Switch: Refuse a change when its before-state cannot be captured (Guardrails › Change safety), on by default. With it on, a write without a before-state does not run, and the review offers manual steps instead. Pull requests are exempt (the repository keeps the before-state).
• Back up a stateful target before a reboot-type change, on by default: before a reboot or restart of an EC2 instance or Azure VM the platform takes an AMI or an OS disk snapshot. Without a usable backup the change does not run. Restoring from it is a rollback a person releases.

Check it: on the change, open Rollback plan. Pre-state per step lists the captured state of each step. The summary reads, for example, 2/2 pre-states. Before / after state lists the reads before and after each step.

If not: a change refused for a missing before-state usually means the write connection cannot read the resource. Test the write connection (section 8), then review the change again. Do not turn the switch off to get one change through.

5.10 Validation: did the fix work?

After a release the platform checks the result, code first and agent last:

1. Post-change validation rules that match the change (metric rounds; three rounds, 300 seconds apart by default).
2. The typed fix's own read about 30 seconds after the write (Check a typed cloud fix with its own read first, on).
3. An agent check only when it can learn something new (Skip agent checks that cannot learn anything new, on).

A verdict reached without an agent is badged platform verdict.

When: after every release. Who: reading: anyone. Verify now: incidents.act.

Steps

1. Open the change. Open Validation plan.
2. Read the verdict: cleared (fixed), still present (not fixed) or uncertain.
3. Only if you need a fresh check, select Verify now. This always starts a paid agent run.

How to check: the stage bar Verify step reads cleared, and the change is completed.

If not:

• still present: the change is failed. The rollback sweep decides what happens next (5.11). The incident reopens one severity level higher (Reopen a failed fix one severity level higher, on).
• uncertain for 24 hours: an approver may select Close without verdict and give a reason. The change becomes failed with verdict uncertain. Nothing rolls back on its own; Roll back stays available.

5.11 Automatic rollback

A failed change is picked up by the rollback sweep within the next scheduler tick, for changes up to 24 hours old. The sweep applies a fixed policy. It never invents a rollback.

The switch Roll back a failed cloud fix automatically (Guardrails › Change safety) is Off — a person releases the rollback by default. Options: L1 changes, L1 and L2 changes. L3 always needs people.

After a rollback lands (or the sweep found nothing to undo), the incident or finding is analysed again with its earlier attempts in view, at most twice (Re-analyse after a failed or rolled-back fix, Guardrails › card 3 · Automatic fixes › fold Demotion, re-analysis and routes, on). The new plan still needs approval. In 0.1.184 that re-analysis appears on the incident with its own trigger.

Check it: the change's Timeline shows the sweep's decision (rolled back, skipped, waiting or escalated) and its reason.

5.12 Roll back a change by hand

When: a change failed and the sweep handed it to a person, or a change that passed validation still caused harm. Who: approvals.decide releases the rollback. Its tier rules apply (L3 needs two people). incidents.act verifies the restore.

Steps

1. Open the change. Or, on the incident, find Undo the executed change and select Review and roll back.
2. Open Rollback plan. Read the stored before-state and the exact reversed edit or command.
3. Select Roll back (the filled button when it is the next step, otherwise under More actions).
4. In the dialog Roll back CHG-…?, read the consequence and select Roll back.
5. Wait for the rollback to finish. An operator then selects Verify restoration only.

How to check: the Timeline shows the rollback receipt. Before / after state shows the resource back at its before-state. The restoration check passes.

If not:

• The rollback outcome is uncertain: select Reconcile rollback outcome. Do not retry the write.
• A restore from a backup (AMI, disk snapshot) loses data written after the backup. Agree that with the service owner first.
• If the change ran through a certified runbook, the rollback may demote that runbook. See 6.3.

API: POST /api/aiops/{tid}/changes/{id}/rollback (approvals.decide), then POST …/rollback-verify (incidents.act).

More detail: Change windows, emergency changes and rollback · Changes reference.

6. Runbooks

A runbook is a fix your team trusts, written as ordered steps: check, backup, change, validate, rollback. The targets are parameters bound from the incident, never one fixed resource. Once an approver certifies a runbook, a fix that follows it exactly is a Standard change: it needs no per-change approver.

Where:

• Changes › Runbooks (/portal/runbooks): every runbook as a card, grouped Taken off pre-approval, Certified, Not certified yet (folded) and Turned off (folded). Tiles at the top: Certified · may run as standard, Taken off pre-approval, Not certified yet, Runs recorded. The folded Runbook catalog has a search box. The folded Platform step templates lists the shared steps.
• Legacy remediation routes (/portal/datasources/legacy-routes; a runbook's name or its Certify / Re-certify link opens it): where a runbook is written, checked and certified.
• Home › Needs you (/portal/approvals): chips Certify (a revision waits for an approver), Demoted (a runbook lost its certification), Not run (a certified runbook matched but did not run).

Who:

How to write a runbook step by step: Write, verify and certify a runbook. The runbooks the platform ships: Runbook library.

6.1 Certify a runbook

When: a new runbook is written, or an operator asked for certification (Needs you › Certify: … asked to certify revision …). Who: an approver (approvals.decide) who did not write or edit this revision.

Steps

1. Operator (writer): on Legacy remediation routes, open the runbook, select Check runbook with a sample alert value and fix every quality-check finding it lists. For a saved runbook, select Check the write connection's permissions.
2. Operator: tick Pre-approved (L1 auto-fix) and select Save route. For an operator this is a request, not a certification. The save reads Certification requested — an approver decides on Needs you.
3. Approver: open Home › Needs you, find the Certify row, and open it. Or open Changes › Runbooks and select Certify on the card. The dialog Certify route '…' opens.
4. Approver: read what you sign: Revision, Mode, Connection, Selector, Tier, Change window, Operations, the Parameters (credentials hidden), the run history of this revision (This revision: N released · N verified by the platform · N failed · N rolled back) and who last changed it.
5. Approver: write the reason in Why is this route safe to run without a per-incident click? (required).
6. Approver: select Certify revision 433c90b5 (the button names the first 8 characters of the route's current revision).

How to check: the card moves under Certified and reads Certified by <approver> · <date>, with the revision. The Certify row leaves Needs you. The audit log has the certification with your note.

If not:

To take a certification back: in the same dialog, select Withdraw certification. The runbook then waits for a person every time.

API: GET /api/aiops/{tid}/routes (read revision), then POST /api/aiops/{tid}/routes/{route_id}/certification with {"action": "certify", "revision": "<revision>", "note": "…"} or {"action": "withdraw", "note": "…"}.

6.2 The four-eyes rule

The server enforces these rules, whatever screen or API is used:

• Only approvals.decide certifies. Editing (datasources.manage) never certifies. The built-in operator cannot certify at all.
• A person who edited the runbook since its last certification cannot certify it while another approver exists. Every editor since the last certification counts.
• An approver certifies the exact revision they read. Any later change to the name, selector, mode, connection, parameters (the steps and plain-words text included), tier or lifecycle flags drops the certification. Needs you lists the runbook again.
• In a workspace with only one approver, that approver may certify their own revision. The dialog says You are the only approver in this workspace: this certification is recorded as self-certified. and the card shows a self-certified badge.

Operator advice: keep at least two approvers in every workspace that uses runbooks. A self-certified runbook is allowed but weaker evidence for an audit.

6.3 Re-certify after a demotion

A certification is taken away automatically, and audited, when:

A fix that did not hold is not held against the runbook when a recorded outside change on the same resource explains the recurrence. The certification is then kept, or restored after the cloud audit log arrives.

After a demotion:

• The card moves under Taken off pre-approval and reads, for example, Demoted after a failed validation <date> — automatic runs stay off until an approver certifies it again.
• Needs you lists it with the chip Demoted.
• Change requests that rested on it and have not run read Normal — runbook withdrawn. New fixes are Normal (Emergency at P1). Nothing runs on its own.

When: a Demoted row is on Needs you. Who: read the cause: anyone. Re-certify: approvals.decide. Keep it manual: a route editor.

Steps

1. Open the change named in the demotion note (A run through it (CHG-…) failed its validation, was rolled back or made a fix that did not hold). Read its Validation plan, Rollback plan and Timeline.
2. Decide with the runbook owner:
   • The runbook is wrong: the writer edits it on Legacy remediation routes, runs Check runbook, saves, and asks for certification (6.1).
   • The runbook is right and the failure had another cause: go to step 3.
   • You do not want it to run without a person any more: select Keep it manual under the route's demotion note. It leaves Needs you.
3. Approver: on Changes › Runbooks, select Re-certify on the card. The certify dialog opens on the current revision. Follow 6.1 steps 4 to 6. A different approver from the editor is needed (6.2).

How to check: the card is back under Certified with a new Certified by line. A new fix that follows it is classed Standard again.

If not: a change request that became Normal — runbook withdrawn while the runbook was demoted stays Normal. Approve it as a Normal change (5.6) or re-analyse the incident for a new request. Also, a fix that did not hold twice in 30 days blocks the runbook on that incident key whatever its certification (Automatic fix blocked on the incident).

Details: Demotion.

6.4 What makes a change Standard

A fix is classed Standard only when all of these hold:

1. The workspace Change class policy is Derived (Guardrails › card 1 · Approvals & change classes).
2. Standard change for a runbook route's operations is on (default on; Guardrails › card 1 › fold How runbooks set the class).
3. A runbook is enabled and certified at its current revision by a named approver. Certified (certifier unknown) does not count.
4. The runbook's selector matches the incident's alerts, and the fix's steps match the runbook's steps, step for step, with every target bound from the incident.
5. Every operation of the fix is pre-approved. A certified runbook owns its own backup and change operations (default on), so you do not need to list them again under Pre-approved operations.
6. No step of the analysis's plan was left out.

One runbook covers one fix. Covering one fix with several runbooks is off by default.

A Standard change is still validated, still captures its before-state, still respects its maintenance window, and still has a rollback plan.

A Standard change still needs approvers when it is at tier L3, when it was not raised from an incident, or when a runbook skill (not a certified runbook) made it standard on a P1 or critical incident.

Check it: on the change, open Class, tier and scope. The class reads Standard — runbook route and the tier line reads pre-approved (certified runbook). The incident's Approval line reads Not needed — pre-approved standard change.

If a change you expected to be Standard is Normal: open the runbook's card on Changes › Runbooks. Look for Changed since it was certified, a demotion, Not certified or Turned off. Also check Needs you › Not run: it names why a matching runbook did not run, for example The analysis proposed typed steps instead of this route. Reasons and fixes: Why a route did not run.

6.5 Run a standard change

When: an incident's fix is a Standard change waiting to run. Who: incidents.act (owner, operator). No approver is asked.

Steps

1. Open the incident. In the What you need to do panel, read the prepared change: the runbook name, its certification, the bound targets and the steps.
2. Select Run standard change. If the window of a live-infrastructure change is closed, the button reads Run standard change — runs in the next window (<time>).
3. In the review dialog, check the exact operation and the remediation connection, then confirm with Run standard change (or the window label).

From the change page the same button opens the same review.

How to check:

• Window open: the change runs. The approvals list and the change request document show Pre-approved standard change — runbook … certified by … at … instead of an approver's vote.
• Window closed: the change reads Pre-approved · scheduled for … and runs when the window opens, after the same checks.
• Afterwards, open Runbook steps for each step's receipt, Rollback plan for the captured before-state and Validation plan for the verdict.

If not:

• The button is missing and an approval is asked: the change is no longer Standard (6.4). Treat it as a Normal change (5.6).
• Not run at its scheduled time, with The change class dropped from standard to normal after this release was scheduled (…): the runbook was demoted or disabled after you scheduled it. Its approvals no longer count. Review it again as a Normal change.
• The change fails validation: the rollback sweep and the demotion rule apply (5.11, 6.3).

6.6 Runbooks that run on their own

A certified runbook can also run with no click at all. This is off by default and needs two switches on Guardrails › card 3 · Automatic fixes: Allow pre-approved routes to auto-fire (the master switch, and the live kill switch) and Allow certified API runbooks with a verified precedent to run on their own. Even then a compliance finding, scanner finding, vulnerability or security alert never runs without a person, a P1 or critical incident never runs on its own through this lane, and L3 is never automatic.

To stop all automatic fixes at once, turn Allow pre-approved routes to auto-fire off. See section 7 for the other safety controls, and Automatic API fixes for every guard.

7. Safety controls

This section covers the controls that stop or limit the agents: the pause switch, the spending limits, the automatic-fix switches, the cap on automatic analyses during a scanner burst, and the defaults you should know before you change anything.

Almost everything here is on one page: Settings › Guardrails & cost (/portal/guardrails). The page has four numbered panels: 1 · Approvals & change classes, 2 · Maintenance windows, 3 · Automatic fixes and 4 · Cost. Below them are the Change safety and Triage & agents panels. The rest is folded at the bottom.

Who can change these controls:

Members without members.manage see a permission notice where the cards would be. They do not see the AI budgets & kill switch card at all.

The full card-by-card reference is the help chapter Guardrails and cost limits. This section gives you the procedures.

7.1 Stop the agents now (Pause all agents)

When to do it. An agent behaves in a way you do not understand. Spend rises fast. A model gateway returns bad answers. You are about to change the execution mode. You are about to do platform maintenance and want no new runs to start.

Who. A workspace owner (members.manage).

What it does. No new agent run starts in this workspace: no analysis, re-analysis, validation check, chat turn, scan or automatic fix. Alerts and incidents keep arriving and are recorded. An incident that would have started a run fails that run and escalates to a person. The pause is checked when a run starts and again at every step, so a run that is already working stops at its next step, without writing an analysis. An automatic cloud rollback waits for a person while agents are paused.

Steps.

1. Open Settings › Guardrails & cost.
2. Go to panel 4 · Cost ("Spending limits") and find the card AI budgets & kill switch.
3. Select Pause all agents (the red button).
4. To stop one run without pausing the whole workspace, open its incident and select Stop this run. Through the API: POST /api/aiops/{tenant_id}/investigations/{id}/cancel.

How to check it worked.

• The card shows Agents are PAUSED — incidents keep opening and escalate to humans. and the button now reads Resume all agents.
• Insights › Agents activity shows Running now falling to 0 as current runs stop.
• The audit log (Settings › Audit log) records the change as Guardrails changed.

To resume. Select Resume all agents on the same card. Incidents that escalated while you were paused stay escalated. Start their analysis by hand with Analyse or Re-analyse on each one.

If it did not work.

• You do not see the card: you do not hold members.manage. Ask a workspace owner.
• A run keeps going: it stops at its next step, which can take as long as one slow model call. On a private-network agent the stop reaches the runner at its next checkpoint. If it still runs, use Stop this run on that incident.
• Several workspaces are affected: the switch is per workspace. Pause each one, or ask platform staff. An expired licence pauses every workspace in the same way (see section 13).

7.2 Set the spending limits

There are four limits. All are in US dollars. 0 means off.

Spend counts operating runs only. Test Lab runs are never counted against these limits.

When to do it. When a workspace is first set up. Before you turn on any switch that starts runs on its own (for example the backlog catch-up in 7.4). After the monthly cost review (section 16) if spend is higher than agreed.

Who. A workspace owner (members.manage).

Steps.

1. Open Settings › Guardrails & cost, panel 4 · Cost.
2. On AI budgets & kill switch, enter Daily limit (USD), Monthly limit (USD) and Per-incident limit (USD). Use whole dollars; 0 turns a limit off.
3. Select Save limits.
4. Open the fold Personal data hiding (PII), cost cap & retention near the bottom of the page.
5. On Cost cap & PII patterns, set Cost cap per investigation (USD, 0 = uncapped). Leave it empty to follow the platform default. The placeholder shows the default in force, for example platform default (10 USD).
6. Select Save guardrails.

How to check it worked.

• AI budgets & kill switch shows Saved. and the values you entered. Spend today and This month show the current spend.
• Cost cap & PII patterns shows Guardrails saved. Reload the page: the cap stays.
• When a daily or monthly limit is reached, the card shows budget tripped — new runs blocked.

If it did not work.

• Spend shows $0.00 even though runs happen: the model is not priced. Every cap is then inactive, because the platform cannot count money it cannot price. Ask platform staff to price the model on Platform defaults → Model price map in the platform console. Until then, do not report $0.00 as a real figure.
• A limit tripped and you need runs now: raise the limit and select Save limits. Runs blocked while the limit held do not restart on their own; start them with Analyse or Re-analyse.
• An estate x-ray scan does not follow these caps. Each scan has its own budget, entered when it is launched.

The platform default for the per-investigation cap is set by your platform team (AIOPS_MAX_COST_USD_DEFAULT). The Helm chart ships 0 (uncapped); the customer example values set 10. Ask which one your deployment runs.

7.3 Automatic-fix switches

By default, no fix runs without a person. Two switches change that, both off when a workspace is created, both on panel 3 · Automatic fixes ("What may run without a person").

The fold Demotion, re-analysis and routes on the first card holds four more switches that decide what happens after a fix. Their defaults are in 7.6.

When to do it. Only after your change process has agreed which runbooks may run on their own, and an approver has certified them (see section 6).

Who. A workspace owner (members.manage).

Steps to turn automatic fixes on.

1. Make sure a daily limit is set (7.2).
2. Open Settings › Guardrails & cost, panel 3 · Automatic fixes.
3. Tick Allow pre-approved routes to auto-fire. The card says Auto-remediation armed.
4. Only if agreed: tick Allow certified API runbooks with a verified precedent to run on their own. The card says Saved — certified API runbooks may repeat a verified fix on their own.

Steps to turn them off. Clear the same boxes. Clearing Allow pre-approved routes to auto-fire stops every automatic fix at once. The card says Auto-remediation off.

How to check it worked.

• The audit log records Guardrails changed.
• On an incident where a pre-approved route matched but did not run, the timeline says Pre-approved route matched — not run, with the reason.

If it did not work.

• The boxes are greyed out and read Blocked by computer-assist mode: the workspace is in Computer assist execution mode (card Execution mode under Triage & agents). In that mode the platform writes nothing. Change the mode back to Standard only after you paused agents (7.1).
• A route matched but did not run: open the incident timeline and read the skip reason. Common reasons are that the route is not certified at its current revision, the incident is P1 or critical, it is flapping, it is outside its change window, or the budget is spent.

7.4 The automatic-analysis burst cap (0.1.184)

A scanner pull can bring hundreds of new findings at once. With Analyse P1/P2 findings automatically on (the default), each P1 or P2 finding would start its own paid analysis. The Burst limit caps how many automatic finding analyses one workspace starts in any 15 minutes.

• Shipped default: 10 per 15 minutes. 0 means no cap.
• A finding over the cap still opens. It is held, with the reason automatic analysis paused: burst limit reached for this workspace.
• A held finding gets its analysis when a person selects Analyse on it, or later through the backlog catch-up, if that switch is on.
• The cap also applies to a finding that reopens and would start a fresh run.
• The cap counts only automatic runs on scanner findings. It does not limit analyses of monitoring alerts.

When to change it. Raise it if your scanners routinely bring more than 10 urgent findings at once and you have budget for them. Lower it, or keep the default, for a large estate whose first scanner pull is still to come.

Who. A workspace owner (members.manage).

Steps.

1. Open Settings › Guardrails & cost, panel Triage & agents, card Automatic analysis of P1/P2 findings.
2. Under the two switches, find Burst limit. Enter a whole number from 0 to 10000.
3. Select Save. To go back to the platform default, select Use platform default.
4. If you want held findings to be analysed later without a click, also tick Backlog catch-up: also analyse P1/P2 findings already held (off by default). It analyses held findings oldest first, 3 per minute. Set a daily limit (7.2) first.

How to check it worked.

• The line under the field reads, for example, At most 10 automatic analyses per 15 minutes; past that a finding waits for Analyse or the catch-up sweep. and says whether it follows the platform default.
• After a scanner sync, the sync record counts the held findings as burst_held.

If it did not work. Many findings still start runs: check that you saved a number above 0. Through the API the field is auto_investigate_burst on GET and PATCH /api/aiops/{tenant_id}/guardrails; null returns it to the platform default.

7.5 Change one guardrail safely

When to do it. Any time you change a switch on Guardrails & cost.

Who. A workspace owner (members.manage).

Steps.

1. Read the switch's line first. It says Following the platform default (on) or Set for this workspace, and names the default.
2. Write down the current value. Reload the page first if someone else may have changed it.
3. Change one switch at a time. Most switches save the moment you tick them; a few cards have their own Save button.
4. To undo a workspace value, use Use platform default where the card offers it.

How to check it worked. Settings › Audit log shows Guardrails changed with your name. The switch line shows the new state.

If it did not work. A save refused with an error names the field. A switch that changes behaviour mid-change (for example the canary order) only applies to changes reviewed after you set it.

7.6 Guardrail defaults to know

These are the shipped defaults (verified against app/config.py for 0.1.184). Each switch follows the platform default until a workspace owner sets its own value.

Never configurable (always on). Shown in the fold Always-on floors: the write floor, the credential execution boundary, x-ray read-only, PII scrubbing at capture, untrusted-input fencing and egress control. See The floors you cannot switch off.

Write posture (platform-level). AIOPS_DIAGNOSIS_WRITES=deny: an agent's write during an analysis is refused and becomes part of the fix plan for a person to approve. Platform staff can set pause instead.

The complete list of workspace settings and their API names is in the appendix and in Settings reference.

8. Connecting sources

This section covers how signals and credentials reach the platform: the alert intake addresses and their token, read and write cloud connections, the scanner sync, built-in detection, and the detector function you can run in your own AWS account.

The pages are all in the folded SETTINGS group of the sidebar:

Scanner sync is on the findings list: Compliance › Findings, fold Diagnostics (for engineers) › Scanner sync.

Who can do what:

8.1 Send alerts to the platform

Every alert source has its own address on Alert intake & tokens. The address has the form https://<platform>/webhook/<source>/<workspace-slug>, for example https://aiops.example.com/webhook/alertmanager/acme. The webhook token proves the sender is yours. Send it in the X-Webhook-Token header. One token covers every source of the workspace.

When to do it. When you connect a new monitoring tool, ticket system or scanner webhook.

Who. An owner mints the token and creates the test alert (pillars.manage). Anyone with access to the tool can paste the address.

Steps.

1. Open Settings › Alert intake & tokens.
2. Under All sources, find your tool and copy its endpoint. For Datadog and Dynatrace, open Payload template for … under the address and copy the template too.
3. Paste the address into the tool's webhook settings. Put the webhook token in the X-Webhook-Token header. If you do not have the token, mint one (8.2) — note that minting replaces the old one.
4. Open the fold Intake rules, service aliases and test alert and go to Send a test alert.
5. Pick the source (alertmanager, generic, datadog or dynatrace) and select Create test alert. Send the sample it shows, with your token in the header.

How to check it worked. The test card shows a receipt: Received — incident opened, Received — joined incident … or Received — identical alert already recorded (deduplicated). A line Attributed by … — add a service label means the alert has no service label; add one so real alerts join the right incident.

If it did not work.

• No receipt observed yet: read the response at your alert source. The intake answers 401 for a wrong token, a wrong workspace slug and a disabled workspace alike, by design. Check all three.
• 429: the workspace sent more than its intake limit (1,200 alert items a minute by default, aiops_ingest_rate_per_min). The whole payload is refused; the sender should retry after the delay given.
• The test expired: a test is valid for 24 hours. Create a new one.
• The alert opened an incident under the wrong service: see Intake rules and the service aliases on the same page, and Storms and unmapped signals.

8.2 Rotate the webhook token

There is no overlap period. The moment a new token is minted, the old one stops working, and every sender still using it is refused.

When to do it. On your agreed rotation schedule. At once if the token may have leaked (pasted in a ticket, a chat or a log). When someone who knew it leaves.

Who. A workspace owner (pillars.manage).

Before you start. List every place that holds the token:

• each monitoring tool and ticket system webhook (the Most used address card shows which source sends most);
• the Security Hub forwarding rule, if the guided AWS setup installed it (EventBridge API destination);
• the detector function in your AWS account, if you installed it (8.7);
• anything that reads the rule dictionary with the webhook token.

Steps.

1. Agree a short window with the owners of those senders. Alerts sent during the gap are refused; most tools retry, but check yours.
2. Open Settings › Alert intake & tokens.
3. On the Most used address card, select Mint webhook token. Confirm with Mint token.
4. Copy the token from the card New webhook token — workspace …. It is shown once.
5. Update every sender on your list. For the detector function, re-run its deploy.sh (8.7) and type the new token when asked.
6. Store the token in your secret manager. Do not paste it into tickets or chat.

How to check it worked.

• From one updated source, send a test alert (8.1) and read the receipt.
• In the next hour, watch Incidents › Incidents for new alerts from each source. A source that goes quiet is probably still on the old token.

If it did not work. A sender gets 401: it still has the old token, or the token was copied with extra spaces. You cannot see the old token again; paste the new one. If you lost the new token before storing it, mint another and update every sender again.

8.3 Connect a cloud account for reading

The agents investigate through read connections. A read connection never writes: every mutating call is refused by the write floor (section 7.6).

When to do it. When you onboard a new AWS account, Azure subscription or other system.

Who. datasources.manage (owner or operator).

Steps (AWS and Azure, the guided way).

1. Open Settings › Keyless cloud access, choose the cloud, the account and the region, and follow the CloudShell setup. It creates a read role and shows you the change set before anything is created. The full walk-through is AWS (and Azure).
2. Open Settings › Data sources and select Connect datasource.
3. Under Use this connection for, choose Investigate only (the default).
4. Fill in the role or credential and select Connect and verify.
5. On the connection's row, select Audit perms.

How to check it worked.

• The connection appears under READ · N — agents may use these to investigate.
• Test on its row passes. Audit perms reads least-privilege OK.

If it did not work.

• Audit perms reads over-privileged: scope the credential down at the source, then select Test again.
• Test fails: read the error. A role the platform cannot assume usually has the wrong trust policy or External ID. For several connections into one account and which jobs each one runs, see Read and write roles.

8.4 Connect a write credential for approved fixes

Fixes run only through a connection that is allowed to execute approved plans. Keep it separate from the read connection where you can, so the read credential has no write permissions at all.

When to do it. When your change process agrees that the platform may carry out approved fixes in an account.

Who. datasources.manage to create it. Changing the use of an existing write-only connection, or pointing it at another address, also needs approvals.decide — of the built-in roles, only the owner holds both.

Steps.

1. Open Settings › Data sources, select Connect datasource, and pick the same type as the read connection. Give it a clear name, for example prod-aws-remediate.
2. Under Use this connection for, choose Approved changes only (separate credential).
3. Give the credential only the permissions the agreed fixes need.
4. Tick Allow use for approved plans when you are ready. It starts off.
5. For AWS or Azure, select Save and check access. Without consent the button reads Save remediation credential and saves without the check.

How to check it worked.

• The connection appears under WRITE ONLY · N — … agents never see these.
• Save and check access shows a passing identity and read check. This is never proof of every future write; each action is checked again before it runs.

If it did not work.

• An operator's change is refused with 403: changing the use or settings of a write-only connection needs an owner.
• An approved fix says no write connection is available: open the fix's review. It lists every enabled write connection and its status.

The full procedure is Write connections and approved fixes.

8.5 Scanner sync (Security Hub and Defender)

The platform pulls findings from AWS Security Hub and Microsoft Defender for Cloud through each read connection. By default it pulls every 15 minutes. Each connection runs the pull only while its job Security findings sync is ticked (the default).

When to check it. Daily, as part of the operator routine. After you add a cloud connection. When the findings list looks out of date.

Who. Anyone with report.view reads the card. Sync now needs pillars.manage. The schedule needs members.manage. Both are owner permissions.

Steps.

1. Open Compliance › Findings.
2. Open the fold Diagnostics (for engineers), then Scanner sync.
3. Read the line Last sync … (Sync now / scheduled) · N investigations started and Last automatic pull ….
4. To pull now, select Sync now. It reads the last 24 hours of findings.
5. To change the schedule, choose a value under Pull from scanners automatically: Off, every 5 min, every 15 min, every hour, every 6 hours or daily.

How to check it worked. The card shows a new Last sync time and how many investigations it started. New P1/P2 findings start their analysis on their own, up to the burst limit (section 7.4).

If it did not work.

• A sync is already running: wait. Its report appears on the card when it finishes. Still running on the server means the page stopped waiting; the sync goes on.
• rate limited: select Sync now again in a minute.
• Scheduled sync is off for this workspace: an owner sets Pull from scanners automatically.
• No findings from one account: check that its connection has Security findings sync ticked under Advanced · agent access and automation credentials › Jobs this connection runs, and that Security Hub or Defender is on in that account.
• Many findings held with burst limit reached: this is the cap working. See section 7.4.

8.6 Built-in detection (no alarms of your own needed)

Built-in detection watches AWS resources that have no alarm. It reads CloudWatch metrics, EC2 and RDS scheduled events and AWS Health through your read connection, and, through a read-only kubectl connection, Kubernetes pod and node state. No model decides anything, and nothing is created in your account.

It is off until a workspace owner turns it on. It then runs in shadow for 24 hours: it records what it would have raised, but opens nothing. After that it arms itself.

When to do it. When an estate has resources with no CloudWatch alarms, or as agreed in the onboarding plan.

Who. A workspace owner (members.manage). The card appears on Data sources only when the workspace has an AWS connection.

Steps.

1. Open Settings › Data sources and find the card Built-in detection (AWS).
2. Select Turn on.
3. Optional: set Poll every and Max resources per poll. Leave Warning rules start an analysis off unless you want every warning to cost a run.
4. During the 24 hours, open Would have fired (shadow) and check the list makes sense.
5. To arm before the 24 hours end, select Arm now and confirm. To stay in shadow longer, tick Keep shadow.

To turn it off, select Turn off. The dialog says how many incidents opened by built-in rules will resolve, annotated built-in detection switched off — no longer watched.

How to check it worked. The card shows its mode badge, Last evaluation … ago, the accounts evaluated and the estimated CloudWatch cost per month. After arming, a breach opens an incident with source builtin.

If it did not work.

• The card is missing: the workspace has no AWS connection.
• An account is listed with a warning: its read role is missing cloudwatch:GetMetricData, cloudwatch:ListMetrics or the inventory has not been harvested yet.
• Not covered, and why lists resources the rules cannot watch, with the reason.

Rules, thresholds and their reasons: Built-in detection and Proactive monitoring.

8.7 Run the detector in your own AWS account

The same built-in rules can run in your AWS account instead, as a Lambda function that EventBridge runs every 5 minutes. It uses the platform's own rule code, reads with its own read-only role and posts only firing and clearing alerts. When it reports that it covered an account, the platform pauses its own detection poll of that account.

Use it when your security team prefers that metric reads stay inside the account, or to reduce the platform's own CloudWatch reads.

When to do it. After built-in detection is on and armed for the workspace (8.6), and the account owner agrees.

Who. In the platform: a workspace owner, for the tokens (pillars.manage). In AWS: someone who may deploy a CloudFormation stack in the account (CloudShell works).

What it creates. One Lambda function (Python 3.12, not in a VPC), an EventBridge schedule, an IAM role with read-only actions plus its own state, secret and logs, a DynamoDB table, a Secrets Manager secret holding the tokens, a 30-day log group, and a private S3 bucket for the code. With --cluster, an EKS access entry with AmazonEKSViewPolicy.

Steps.

1. In the platform, open Settings › Synthetic checks, card Automation in your account, and select Create token. Copy the automation token; it is shown once. (If the workspace already has one, use it. Rotate token would stop the automation already installed.)

2. Have the webhook token ready (8.2).

3. In CloudShell in the target account, from the release's deploy/aws/tenant-detector directory, run:

   ./deploy.sh --webhook-url https://aiops.example.com/webhook/generic/<workspace-slug> \
               --region <region> [--cluster <eks-cluster-name>]
   

   The script asks for the webhook token and the automation token. It never takes them on the command line.

4. For the Kubernetes node rule (R34), apply the read-only RBAC: kubectl apply -f k8s-rbac.yaml.

5. Test without sending anything:

   aws lambda invoke --function-name aiops-tenant-detector-detector \
     --cli-binary-format raw-in-base64-out --payload '{"dry_run":true}' /dev/stdout
   

How to check it worked.

• On Settings › Synthetic checks, the Automation in your account card shows Rule detector: Live — last called in … within one schedule interval.
• A firing alert from it carries the label detector=tenant-lambda.

If it did not work.

• Rule detector: Stale: the function has not posted for over 30 minutes. Read its log group in CloudWatch Logs. The platform keeps polling the account itself while the detector is stale, so coverage is not lost.
• Another rule detector (…) was refused: one detector runs per workspace. Remove the second one, or wait 30 minutes after removing the first.
• The platform still polls the account: the function posts its coverage only when it read the platform's rule dictionary and delivered every alert. A missing or rotated token, a failed reader or an undelivered alert keeps the platform polling. Check the function's output.
• The Kubernetes rules are skipped: the cluster endpoint must be public. A private cluster needs the private-network agent instead.

To rotate its tokens. Re-run ./deploy.sh with the same arguments and type the new tokens. To remove it. Run aws cloudformation delete-stack --stack-name aiops-tenant-detector, then delete the code bucket.

Full reference: deploy/aws/tenant-detector/README.md in the release, and Automation in your account.

Next steps

• Next: Operator runbook: the platform team
• All Install and operate the platform pages
• Documentation home