Runbook library and runbook rules

The step runbooks shipped with the platform, where each one comes from, and the rules every runbook keeps — one use case, never one resource.

Reference · Updated 2026-10-28 · Describes release 0.1.164

What this page covers

Runbooks and automatic fixes explains how a step runbook is written, certified and released. This page lists the runbooks the platform ships and the rules that keep a runbook general.

One use case, never one resource

A runbook describes how to fix one kind of problem, for example "IMDSv1 allowed on an instance". It never describes how to fix one particular resource. The quality check enforces this, so a runbook that breaks either rule below cannot be certified:

• Every backup and change target comes from the incident. It is a ${parameter} bound from label:<name>, annotation:<name> or plan. A target written out with no parameter, or a literal parameter that holds a resource id, reads pinned_target. A literal next to a bound parameter is allowed: the Lambda example's alias name in ${function}:${alias}, and the EC2 patch example's isolation group in args.security_group. That group is a tool the runbook uses, not the resource it fixes.
• The selector names the use case. That means a control, a service, a resource type or a pattern. A selector value that is one exact resource id, ARN or Azure resource id reads pinned_selector, and so does any value without a wildcard under a key that names the resource: resource, resource_arn, resource_id, arn, instance, instance_id, resource_name, instance_name, hostname, host, vm, vm_name or name. A pattern such as arn:aws:ec2:*:*:instance/* is allowed. The route editor's preview reads the selector as edited, so the check shows before you save.

Why: a runbook certified on one instance, with that instance written into it, used to produce that instance's change for another instance's incident. A runbook certified with a pinned target before this rule existed now never expands into a change and never matches a fix; the incident records it under Pre-approved route matched — not run with the reason pinned_target on Needs you, and the route list shows the finding. The analysis's own steps on the incident's resource then make a normal change that a person releases.

The check cannot see a service selector whose value happens to be one resource's name, for example a service label equal to a VM's name. Write the selector for the use case.

AWS-published runbooks

AWS publishes remediation documents for Security Hub controls in its open-source project Automated Security Response on AWS (ASR). The platform does not run those documents. Running one installs ASR's own stacks in your account: Step Functions, Lambda functions and one IAM role with write access per remediation. Instead, the platform captures each document's steps as a step runbook made of its own typed operations, so the same write connection, approval, pre-state capture and rollback apply as for every other fix. Five runbooks are captured from ASR v4.1.0, all of them documents whose every write is already a typed operation:

They are added once to every workspace that has an enabled AWS write connection (role write or read/write), bound to the oldest such connection, within one scheduler tick of the connection being enabled — disabled and not certified: the runbook catalog lists them as Not certified · disabled until a person points one at the connection they want and an approver certifies it, and both the catalog row and the route row name where each came from — Published by AWS · ASR-EnableIMDSV2OnInstance · Automated Security Response on AWS v4.1.0 (params.source: publisher, document, project and version); nothing enables them on its own. A workspace with no AWS write connection gets none, and a connection disabled later leaves the routes as they are. Once added they are yours: a runbook you delete or rename is not added again (the audit row route.asr_seeded records the version of the AWS set the workspace received; a later version adds only its new names and never touches an existing route).

Each runbook has the same shape. A check reads the current state before the write, and that recorded state is the backup: these are configuration changes that touch no data, so no image is taken. Then comes the change, then a validate step (the platform's typed read, then the control reading PASSED on Security Hub's next evaluation), and a compiled_inverse rollback that puts the recorded state back. The route's params.source records the AWS document, document_owner, the ASR version (v4.1.0), the controls and any difference from the document. The check step's reason names the document too, so the steps a certifier reads say where they came from.

Differences from the documents:

• ASR revokes every matching rule in one run. These runbooks revoke the one rule the analysis names, one change step per rule. A fix with two rules matches only when the workspace lets several runbooks cover one change.
• The IAM password policy uses ASR's baseline (length 14, every character class, 90 days, reuse 24) but sets HardExpiry false where ASR sets true: an expired password never locks a user out.
• AWS-DisablePublicAccessForSecurityGroup's role may also revoke egress rules and edit rule descriptions. Its description removes public access to the admin ports only, and that ingress revoke is what the runbook captures.

A runbook that takes a target from the analysis (plan) can make a fix a standard change once certified, but it never runs on its own. The EC2.8 runbook binds its instance from the finding. A finding is never fixed without a person, whatever runbook matches, so a person releases all five.

Where they are. The platform seeds them in the demo workspace, disabled and uncertified, on the placeholder write connection example-aws-writer (no credentials). To use one, point it at your AWS write connection, enable it, and have an approver Certify it. Each one passes the quality check. The route editor's permission check lists the actions each step needs:

Not captured yet. These common controls have a document with at least one write that has no typed operation yet: EC2.7 (ASR-EnableEbsEncryptionByDefault), CloudTrail.1 (ASR-CreateCloudTrailMultiRegionTrail: a bucket, its policy and the trail) and SSM.1 (ASR-AttachSSMPermissionsToEC2: an instance profile and its role). A fix for them is a normal change with operator lines.

Package upgrade on an EC2 instance

aws.ssm.upgrade_packages upgrades named OS packages, such as a library a scanner flagged, in place on one EC2 instance. It runs through Systems Manager Run Command with the AWS-owned AWS-RunShellScript document. Like the patch, it is a runbook-only operation, so the analysis never proposes it. Rollback is not the same as backup: a package manager does not downgrade, so the quality check requires an image backup of the same instance before the upgrade and the restore of that image as its rollback.

{"version": 1,
 "parameters": {"instance": {"kind": "aws.ec2.instance", "from": "annotation:resource_arn", "required": true}},
 "steps": [
  {"id": "check", "phase": "check", "target": "${instance}"},
  {"id": "image", "phase": "backup", "operation": "aws.ec2.create_image", "target": "${instance}"},
  {"id": "upgrade", "phase": "change", "operation": "aws.ssm.upgrade_packages", "target": "${instance}",
   "args": {"packages": "openssl openssl-libs"}},
  {"id": "upgraded", "phase": "validate", "criteria": "openssl and openssl-libs each read a different installed version on ${instance}"},
  {"id": "undo", "phase": "rollback", "operation": "aws.ec2.restore_root_volume"}]}

• The packages are the allowlist. args.packages names at most 16 packages by their exact lower-case names (letters, digits, ., +, _, -). It holds no version, path, wildcard or shell syntax. The names are written in the runbook, or in a literal string parameter, so the certifier reads exactly what may be upgraded. A label or annotation never supplies them, because that would let an alert choose what gets installed. The quality check reads packages otherwise.
• The script is the platform's own. The platform compiles one script from the names and sends nothing else. On an instance with apt-get it runs apt-get install --only-upgrade; otherwise dnf upgrade, or yum update where there is no dnf. Each version is printed before and after the upgrade, and the package manager's own output goes to the command's error stream. A package that is not installed stops the script before anything changes, because only installed packages are upgraded. The rendered call is checked against the reviewed names right before it is sent, and any other command, document or instance is refused.
• How it ends. The instance must be an online Systems Manager node. The command is then polled like the patch, for up to 60 minutes. Success with every package reading a different installed version afterwards completes the step, and the receipt lists each package's version before and after (openssl 3.0.8-1 -> 3.0.8-2). A package that reads no version afterwards fails the step, and so does one whose version is unchanged (held, pinned, or no newer build on the mirror: openssl stays at 3.0.8-1 — the finding is not fixed), Failed, TimedOut or Cancelled. A failed step ends the change at execution — no validation verdict, so the runbook keeps its certification — and a person decides whether to restore the image.
• Validation. The validation reads the command's own result again (ssm:GetCommandInvocation, read-only): every named package must read a different version from before. Versions are not compared, so different is what is checked, not newer.
• Rollback. The rollback restores the root volume from the image, released by a person. Section 4 of the change request shows the ssm send-command line, and the rollback plan says the upgrade has no inverse.
• Permissions. The write role needs ssm:SendCommand on the AWS-RunShellScript document and on the instance, plus ssm:DescribeInstanceInformation, ssm:GetCommandInvocation, ec2:DescribeInstances, ec2:CreateImage (the backup) and the restore's ec2:CreateReplaceRootVolumeTask. IAM cannot narrow SendCommand on AWS-RunShellScript to these commands: a role that holds it can run any shell command on the instances it may target. For that reason the guided AWS setup does not grant it, and the permission check reads it missing until you add it to your write role, scoped to the instances tagged aiops-managed=true. What limits the command is the platform: it sends only the compiled script.

Demotion

A certification is not permanent. Two workspace switches on Guardrails (in the product: /portal/guardrails) take it away, each audited and listed on Needs you → Demoted for a route editor to decide:

Both switches follow every route a change ran through: the route's own run, and each runbook a change request followed step for step. A change minted from a step runbook carries no route of its own, and before this release its rollback demoted nothing. A deviation found after an approver edited the runbook and certified the new revision does not demote it: the revision that deviated is already gone. The audit row names who declared the deviation (declared_by: the validation, or the person whose Verify found the condition still present).

A fix that did not hold is not the runbook's deviation when a recorded outside change explains it. Before demoting for a fix that did not hold, the platform reads the change ledger for a write on the same resource, made after the fix and no later than the recurrence, by a principal that is not the platform's own (a harvested cloud write recorded as outside or unknown; a write the cloud refused never counts). With one, the runbook keeps its certification: the audit row Runbook kept — an outside change explains the recurrence names the fix, the outside change and the recurrence incident, the route's certification line adds a fix it made (CHG-…) did not hold — an outside change (CHG-…) on the same resource explains it; certification kept, and the recurrence incident's Timeline carries the same row. The did-not-hold mark on the last-known-good record stands either way: the condition did come back.

The mark and the demotion are written when the recurrence opens, and a cloud audit log delivers the write minutes later (CloudTrail 5 to 15 minutes), so a runbook can be demoted before the write that explains it reaches the ledger. Since 0.1.164, after each audit harvest that records a cloud write, the platform reads the did-not-hold marks of the last 24 hours that name no outside change again. A mark the new write explains gets the audit row Recorded fix recurrence explained by an outside change and no longer counts against the fix. The runbook that this recurrence's mark demoted is certified again with the whole certification it held, audited Runbook certification restored — an outside change explains the recurrence (route.deviation_explained, restored: true) on the recurrence incident; the route's certification line reads as for a kept certification. The recurrence's change requests that nobody has requested yet, classed normal while the runbook was demoted, are classed again (Change class restored — the runbook was certified again). The runbook stays demoted, for an approver to certify, when the demotion came from another recurrence, when the runbook was demoted again since (a failed validation, another unexplained recurrence, a rollback), when a person disabled, edited, certified or withdrew it or chose Keep it manual, or when the demotion was written before 0.1.164.

Deviation is our proposal. Your change process described this rule as one that applies in some cases, not in all of them. The platform turns it on by default. A workspace that wants only rollbacks to demote turns it off (PATCH /guardrails with {"demote_on_deviation": false}; null goes back to the default). It applies only to certified runbook routes. A pre-approved route of another kind keeps the behaviour from before this release.

What you see after a demotion:

• On the route. Demoted after a rollback <date>, Demoted after a failed validation <date> or Demoted after a fix that did not hold <date>, followed by — automatic runs stay off until an approver certifies it again. The note under Pre-approved names the change: A run through it (CHG-…) was rolled back, failed its validation or made a fix that did not hold.
• In the audit log. Pre-approved route demoted after a rollback or Certified runbook demoted after its fix deviated, with the change, the revision and the reason.
• On change requests. A request whose standard class rested on the runbook and that has not run yet reads Normal — runbook withdrawn, and its next fixes are normal changes. Nothing runs on its own.
• On an incident it later matches. Pre-approved route matched — not run with The route was demoted after a rollback on <date> or The runbook was demoted on <date>: a run through it failed its validation (or a fix it made did not hold).

To restore it, an approver certifies the route again (Certify on the routes page). A route editor's tick of Pre-approved is a request that an approver decides (Ticking it again asks an approver to certify this revision.). Keep it manual takes it off Needs you. Separately, a fix that did not hold twice in 30 days blocks the route on that incident key, whatever the switches say (Automatic fix blocked).

Next steps

• Next: Automatic API fixes for proven runbooks
• All Work with the agents pages
• Documentation home