Customer deploy runbook — a fresh AWS account in 8 steps

The managed install into one dedicated AWS account in eight steps, with upgrade, rollback, teardown, costs, licence and secret rotation.

Reference · Updated 2026-10-19 · Describes release 0.1.164

Managed BYOC, profile 1 (decision brief 2026-09-18; decision log §6.4): the customer owns one dedicated AWS account and one IAM role our pipeline may assume. Our Azure Pipelines create the stack in that account through that role and helm upgrade --install our OCI chart. Nothing of ours runs 24x7 in their account beyond the platform itself: no ArgoCD, no licence server, no Kyverno.

Files this runbook drives:

<slug> is one string used everywhere: env dir, values dir, branch customer/<slug>, ADO Environment aiops-<slug>, variable group aiops-<slug>-deploy. The per-customer files carry the customer's account id and host name, so they live on branch customer/<slug> and never on main.

0. The whole install

The host owner is whoever holds the DNS zone of the platform's host name: us for a test host of ours, the customer for their own domain.

Times are those observed in the 6 October 2026 rehearsal on build 0.1.105 (02-install.md): measurements, not an SLA. The step 5 phases overlap and sit inside its interval, so do not add them up.

1. Before you start

1.1 The account

• A dedicated AWS account, or a new member account in the customer's Organization (§1.4 explains why dedicated).
• Ask about SCPs, permission boundaries and mandatory tags before step 2. The bootstrap tags what it creates with Project and ManagedBy only; the stack's tags come from env.hcl (tags).
• Step 2 runs as an account admin: it creates an IAM role, may request an ACM certificate, and reads EC2, Service Quotas and Bedrock.

1.2 Region

• EKS Auto Mode and Bedrock available (or the customer's own OpenAI-compatible endpoint), and two availability zones EKS supports. The bootstrap picks two and skips the AZ ids EKS cannot use (use1-az3, usw1-az2, cac1-az3).
• Images cross regions through the ECR-to-ECR pull-through cache (supported across Regions and accounts); our registry stays in ap-southeast-1.
• Terraform state stays in our bucket: state_region in env.hcl names the bucket's region, so the customer region may differ from it.
• A fresh account or region can have a small On-Demand vCPU quota; the bootstrap warns below 8.
• Proven so far: ap-southeast-1 (§8). Any other region is proven by its first install.

1.3 Model

• Claude through a Bedrock inference profile of AIOPS_ANTHROPIC_MODEL (default anthropic.claude-sonnet-5). The bootstrap picks the geographic profile (apac., eu., us. …). A global. profile routes requests worldwide, a data-residency question the customer answers: it is picked only with AIOPS_ALLOW_GLOBAL_ROUTING=true, and when the region offers Claude only through global., the bootstrap says so and recommends Nova Pro until they opt in. In THIS account:

  1. Anthropic's use-case form is on file — submitted in this account's Bedrock console, or inherited from the Organization's management account.
  2. The model agreement exists — created by one first invocation from the console Playground by an admin allowed aws-marketplace:Subscribe, or by aws bedrock create-foundation-model-agreement (AWS CLI 2.27.42 or later). The bootstrap prints the two commands (list the offer, then create the agreement) and never runs them: accepting the model's terms is the customer's decision. A PENDING agreement is waited for, never created twice.
  3. The rest of Bedrock's availability answer allows it: the account is entitled to the model (AWS can withhold Anthropic models from an account), IAM and SCPs authorise it, and the region serves it. The bootstrap recommends Claude only when all four fields allow it, and otherwise names the field that does not in model.anthropic_next_step.

  The platform's Pod Identity roles hold bedrock:InvokeModel* only and cannot create the agreement. While Claude is not available to the account, every investigation fails on the runner with … is not available for this account.

• Nova Pro (bedrock/<geo>.amazon.nova-pro-v1:0) otherwise: the bootstrap's model.recommended picks it automatically when Claude is not available, and render copies it into AIOPS_MODEL.

• Their OpenAI-compatible endpoint: render --openai-endpoint, then set AIOPS_MODEL=openai/<model>, AIOPS_API_BASE and AIOPS_API_KEY in the secrets file before ado.

Check before booking the install date: on 2026-09-28 AWS refused Anthropic models for our test account even after the form ("contact AWS Sales").

1.4 What the deploy role can do

aiops-deployer holds one inline policy, aiops-deployer-policy.json: what the _customer-template units create, unit by unit (checked against the module sources: vpc 6.0.1, security-group 5.3.0, eks 21.0.6, rds 6.10.0, vpc-endpoints 6.0.1, and the local ecr-cache / iam / alb-access / cost-guardrails tf).

Said plainly: IamForTheStack allows iam:AttachRolePolicy, iam:PutRolePolicy and iam:CreatePolicyVersion on role/aiops-* and policy/aiops-* with no iam:PolicyARN condition, and aiops-deployer itself matches role/aiops-*. The role can therefore attach any policy, AdministratorAccess included, to itself: in practice it can make itself administrator of the account. The permissions boundary described in the 18-Sep decision brief (§3.1) is not built. Containment today is the dedicated account, plus the fact that only our pipeline can assume the role (our infra role, the ExternalId we minted, and a SourceIdentity naming the operator), so every session and every call lands in the customer's CloudTrail under that name. The boundary is the hardening before any production install, or before a test if the customer asks for it.

The trust (aiops-deployer-trust.json) allows our infra CI role only: sts:AssumeRole with the ExternalId, a Deny on sts:AssumeRole without a SourceIdentity, and separate sts:SetSourceIdentity and sts:TagSession statements. The ExternalId key exists only in AssumeRole's request context — on a statement that also lists SetSourceIdentity or TagSession it never matches, and STS refuses the whole call. Access Analyzer reports no findings on this document. Our side of the chain is the AssumeCustomerDeployRole statement in infrastructure/iac/oidc-role/infra-role-policy.json.

1.5 Host name and certificate

• A test: our test host name, so the customer has no DNS work. The bootstrap requests the certificate in their account (AIOPS_HOST), and we add both CNAMEs (validation, then the load balancer) in our zone.
• Production: their host name. They add both CNAMEs, or bring an ISSUED certificate of their own (AIOPS_CERT_ARN).
• DNS-only records: a proxying CDN in flexible-TLS mode loops on the ALB's ssl-redirect. No external-dns, no cert-manager.

1.6 Network

• The stack keeps a NAT gateway: the first pull through the cache rule, webhooks and tickets need it. A fully private VPC switches image delivery to ECR replication, which is not built.
• A path to their estate (peering / TGW / PrivateLink), or the private-network agent (deploy/aiops-agent-helm) in their VPCs. render --estate-cidr opens the runner's egress to their private estate.

2. Install, step by step

Step 1 — Vendor: mint

python3 scripts/customer-onboard.py mint <slug>

Writes ~/.aiops/customers/<slug>.env (mode 0600) once and never overwrites it: AIOPS_DEPLOY_EXTERNAL_ID (a UUID: the confused-deputy guard, minted by us per customer), SECRET_KEY, ADMIN_PASSWORD, AIOPS_SECRET_KEY (the Fernet key of the tenant secret store: the file variable reaches the pod as AIOPS_TENANT_SECRET_KEY; losing it loses every stored datasource credential, so back the file up like a key; rotating it is §12.1) and empty AIOPS_SECRET_KEY_PREVIOUS, AIOPS_API_KEY, AIOPS_MODEL, AIOPS_API_BASE. It prints the hand-off: the zip command (git archive of bootstrap.sh and the two JSON files, run from a clean checkout of main) and the sha256sum of the same three files, the ExternalId, and the customer's command line. Send the zip with the sums, and the ExternalId in a separate message. A second mint prints the same hand-off again.

Step 2 — Customer: bootstrap

In AWS CloudShell opened in the install region, as an account admin:

unzip aiops-bootstrap.zip
export AIOPS_ACCOUNT_ID=111122223333 AIOPS_EXTERNAL_ID=<the ExternalId from the vendor>
export AIOPS_HOST=aiops.example.com      # optional: request the certificate here
bash deploy/customers/bootstrap.sh plan  # read-only
bash deploy/customers/bootstrap.sh apply

An account with only its root user. plan stops when the caller is the account root (or a federated session): EKS access entries take an IAM user or role, and root cannot assume a role, so root cannot be the cluster's break-glass admin. Create a role for the people who will use it and name it before plan:

cat > break-glass.yaml <<'EOF'
Resources:
  BreakGlass:
    Type: AWS::IAM::Role
    Properties:
      RoleName: platform-break-glass
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: sts:AssumeRole
      Policies:
        - PolicyName: eks-describe-cluster
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: eks:DescribeCluster
                Resource: "*"
EOF
aws cloudformation deploy --stack-name platform-break-glass --template-file break-glass.yaml \
  --capabilities CAPABILITY_NAMED_IAM --no-fail-on-empty-changeset
export AIOPS_ADMIN_ROLE_ARN=arn:aws:iam::$AIOPS_ACCOUNT_ID:role/platform-break-glass

The deploy makes that ARN a cluster admin through an EKS access entry (render copies the bootstrap JSON's admin_principal_arn into eks_admin_principal_arns); the role's own policy only lets aws eks update-kubeconfig read the cluster. Its trust names the account, so an IAM user or SSO role of yours that may call sts:AssumeRole on it can use it: give at least one person such an identity, since root cannot. Never name it aiops-*: the platform's control-plane and runner roles may assume any role so named, and in this account a trust that names the account is all they need.

apply creates or updates the role aiops-deployer from the two JSON files (trust rendered with our pipeline role and the ExternalId, 1-hour sessions, tagged ManagedBy=aiops-bootstrap), and writes only what differs: a second apply makes no change. It refuses an existing aiops-deployer it did not create and whose trust does not name our pipeline role. plan and apply also check the availability zones, the vCPU quota and Bedrock access, and print one JSON (saved as ./aiops-bootstrap-<account>.json) that holds no secret — never the ExternalId:

account_id, region, azs, deploy_role_arn, deploy_role_ready, admin_principal_arn (goes into the cluster's access entries, so someone of theirs can always kubectl in), host, certificate_arn, certificate_status, validation_record, model (recommended, anthropic_agreement, use_case_form, anthropic_next_step, claude_profile, nova_profile), bedrock_in_region, ec2_standard_vcpu_quota, existing_vpc_cidrs, policy_sha256.

A read it cannot make (the vCPU quota, Bedrock) is printed as a note: line with the AWS error instead of being dropped. They send back the file apply wrote. A plan file says deploy_role_ready: false until the role exists, and render refuses it.

Step 3 — Certificate validation

The host owner adds the validation CNAME from validation_record (DNS only) and waits until ACM shows ISSUED; re-running bootstrap.sh plan prints the status. The ALB gets no HTTPS listener until the certificate is ISSUED.

Step 4 — Vendor: render, Azure DevOps, pull grant

git switch -c customer/<slug>
python3 scripts/customer-onboard.py render <slug> aiops-bootstrap-111122223333.json \
  [--estate-cidr 10.20.0.0/16] [--openai-endpoint] [--alarm-email aiops-oncall@example.com]
git add infrastructure/iac/envs/<slug> deploy/customers/<slug>
git commit -m "chore(deploy): add <slug>" && git push -u origin customer/<slug>
ADO_PAT=<a PAT of yours> python3 scripts/customer-onboard.py ado <slug>

• render copies _customer-template to infrastructure/iac/envs/<slug>/ and fills env.hcl (region, account, our state bucket, deploy role, name prefix and cluster name aiops-<slug>, the Customer tag, the AZs, eks_admin_principal_arns, enable_bedrock_endpoint = false for --openai-endpoint or a region without Bedrock, and alarm_email — the RDS alarms' named recipient, a role mailbox since it is committed — from --alarm-email; the address confirms the subscription mail AWS sends after the deploy). It fills deploy/customers/<slug>/values.yaml from the example (image repository, host, certificate, public URL, the keyless trust anchor role and account, region, runner role, the VPC and estate CIDRs; without --estate-cidr the estate rule repeats the VPC CIDR) and sets AIOPS_MODEL in the secrets file from model.recommended. It runs the deploy's own placeholder guard and refuses: off branch customer/<slug>, a plan JSON, a JSON without host or certificate, any value not of its exact shape (terragrunt evaluates env.hcl with our pipeline credentials, so the customer's file is checked field by field), no usable model without --openai-endpoint, and files it rendered before unless --force (a re-render drops hand edits such as a changed vpc_cidr; read git diff before committing). It warns when the VPC CIDR overlaps one of theirs, the certificate is not ISSUED yet, or their deploy role carries another version of the policy than this commit. The RDS endpoint is not filled in: the deploy reads it from the instance. The ExternalId never lands in git: deploy_external_id reads it with get_env().

• ado (a human runs it: it grants) registers deploy-customer.yml and destroy-customer.yml if missing, creates Environment aiops-<slug>, creates variable group aiops-<slug>-deploy from the secrets file, and authorises both pipelines on the group, the Environment and the aws-oidc-infra service connection. It reads ADO_PAT from the environment only. Re-running it creates nothing new and leaves an existing group as it is. The secrets file is the source of truth: to change a value, edit the file and run ado <slug> --update-group, which rewrites every value of the group from the file (a value changed only in ADO is lost).

  Variable	Secret	Meaning
  AIOPS_DEPLOY_EXTERNAL_ID	yes	the ExternalId in their trust policy
  SECRET_KEY	yes	secrets.AIOPS_SECRET_KEY — signs only the 10-minute SSO sign-in state and the 24-hour SSO logout hint. Sessions are stored hashed in the database, so rotating it signs nobody out; an SSO sign-in in progress restarts
  ADMIN_PASSWORD	yes	secrets.AIOPS_ADMIN_PASSWORD — bootstrap superadmin
  AIOPS_SECRET_KEY	yes	secrets.AIOPS_TENANT_SECRET_KEY — Fernet key of the tenant secret store
  AIOPS_SECRET_KEY_PREVIOUS	yes	secrets.AIOPS_TENANT_SECRET_KEY_PREVIOUS — the store key being rotated out (§12.1); empty otherwise
  AIOPS_API_KEY	yes	LLM gateway key; empty on Bedrock
  AIOPS_LICENSE	yes	licence.value — the offline licence JWT (§11); empty = unlicensed
  AIOPS_MODEL	no	bedrock/<inference-profile-id> or openai/<model>
  AIOPS_API_BASE	no	the OpenAI-compatible base; empty on Bedrock

  The deploy refuses an empty AIOPS_MODEL (that would be the mock LLM), SECRET_KEY, ADMIN_PASSWORD or AIOPS_SECRET_KEY, and an AIOPS_SECRET_KEY_PREVIOUS equal to AIOPS_SECRET_KEY.

• Blocking: add an approver under Pipelines → Environments → aiops-<slug> → Approvals and checks. An Environment created by ado has no check, and both the deploy and the destroy would then run unattended.

• Pull grant, on main: add the account id to platform_image_pull_account_ids in infrastructure/iac/envs/<host-env>/env.hcl (<host-env> = our sandbox host environment), then run infra-terragrunt.yml with env=<host-env>, action=plan and then action=apply, for unit=ecr and again for unit=ecr-charts. Removing the id and re-applying is the per-customer revocation lever: new versions stop, the version already cached keeps running. Azure Repos runs no PR plan, so the manual plan is the check.

Step 5 — Vendor: deploy

Run deploy-customer.yml from branch customer/<slug> with customer=<slug>, imageTag=<version> (a tag build-ecr.yml pushed — plain 0.1.101 or 0.1.101-<suffix>.<buildId> with the pipeline's fixed build-tag suffix; empty = the version in pyproject.toml) and bootstrapDb=true, in a window booked with the customer.

• Stage Plan checks the values file for placeholders and shows the terragrunt plan. On a first install that is the whole stack; there is no separate infra run.
• Approve on aiops-<slug>.
• Stage Deploy applies terragrunt (on a first install this creates the whole stack: VPC, EKS, RDS, image cache, VPC endpoints), resolves image digest + chart, verifies the signature (or warns while signing is off), snapshots RDS, writes kubeconfig, applies deploy/customers/cluster-prereqs.yaml (IngressClass alb + default StorageClass gp3 — Auto Mode ships neither, and without them the Ingress is ignored and the PVC never binds), reads the RDS endpoint + master secret ARN from the instance, and runs the in-cluster psql bootstrap (CREATE ROLE aiops LOGIN; GRANT rds_iam TO aiops; + database + schema grant; the master password is read from the RDS-managed secret into the pod env and never printed; the pod may wait up to 5 minutes for the cluster's first node).
• Then helm upgrade --install aiops oci://…/charts/aiops --atomic --wait --timeout 10m with the values file, image.tag, the RDS endpoint and config.AWS_REGION from env.hcl. It waits up to 3 minutes for the ALB, prints CNAME target: <host> -> <alb>, and polls https://<host>/healthz through the ALB under the real host name (SNI and the ACM certificate still checked, so it passes before the CNAME exists) until it answers {"ok":true,"version":"<version>"}.
• It publishes deploy-log.json (image, digest, chart, snapshot, host, ALB name, commit). The EKS API endpoint is public (IAM-authenticated) because helm runs from a hosted agent.

Step 6 — DNS

The host owner creates ONE CNAME from the host name to the printed ALB name (CNAME target: …, also alb_dns in deploy-log.json), DNS-only. The deploy gate warns while public DNS does not reach the platform yet.

Step 7 — Vendor: price map and smoke

• Model price map: in /ops → Platform defaults (/ops/platform) → Model price map, press Fetch list prices. A Bedrock model is priced from the public AWS Price List for the region it is called from — through the gateway's aws_region_name on that entry, or for a direct bedrock/… AIOPS_MODEL (e.g. bedrock/apac.anthropic.claude-sonnet-5, the regional price) the control plane's AWS_REGION — and the result says per model what was recorded or why not. Every other model is priced at its publisher's list price from models.dev (the publisher's own entry only): the reviewed router table maps cx/, glm/, ds/ and cc/ aliases; anything else is mapped from Needs attention → Map. A customer contract or discounted rate is a manual entry with its source note (it outranks the list price). A workspace on its own endpoint shows "Not priced yet" until its models are mapped under Workspace endpoints — tell the customer before the upgrade that a rate card they typed stops pricing new runs. Check Needs attention is empty for the smoke's model: with no trusted entry, a model id litellm already prices books its bundled table's figure, labelled an estimate on every cost screen; an id it does not know (a gateway alias) books $0, the cost guardrails (per-run cap, tenant-day budget) can never fire, and the smoke fails on cost_usd=0. api-mode runners pick the card up on their next job; no restart. The daily sync needs outbound HTTPS to pricing.us-east-1.amazonaws.com and models.dev (control plane only; runners never call either); without them set AIOPS_PRICE_SYNC_INTERVAL_S=0 (or only AIOPS_MODELS_DEV_PRICES=false for the models.dev part) and record manual prices. To reconcile tokens later: python -m app.aiops.price_sources reconcile --days 7 in the control-plane pod (needs cloudwatch:GetMetricData), or Cost Explorer by usage type in the customer's account.
• Smoke, from any machine that reaches the host: BASE=https://<host> TENANT=<first-workspace-slug> ADMIN_PASSWORD_FILE=<file> scripts/customer-smoke.sh. It logs in as admin@local, creates the workspace (owner admin@local) if missing, mints the webhook token (printed ONCE — pass WEBHOOK_TOKEN= on any later run, minting replaces the old one), fires one generic alert, waits for the investigation to finish, and fails unless a REAL model answered (per-model usage recorded, cost_usd > 0) with a non-empty RCA; it prints the incident URL. In remote mode the control plane never runs an investigation, so done here proves the runner plane, the runner's model access and region. There is no delete-workspace endpoint: use the customer's real first workspace slug. The minted token is a live ingest credential: store it in the customer's secret store straight away, and never paste the smoke output into a ticket, chat or email — share the incident URL only.

Step 8 — Customer: first login and first connection

1. Log in at https://<host> as admin@local with ADMIN_PASSWORD, handed over out of band (never in the same message as the URL), and change it: /ops → Users → Reset password.

2. Connect their identity provider under Settings · Account → Members & roles → Single sign-on (Single sign-on), then point their alert sources at https://<host>/webhook/<alertmanager|cloudwatch-sns|azure-monitor|…>/<slug> with the token as Authorization: Bearer <token>.

3. Keyless AWS (the first read-role datasource): both Pod Identity roles (aiops-<slug>-aiops, aiops-<slug>-aiops-runner) carry sts:AssumeRole + sts:TagSession on any account's roles named aiops-* (a role named otherwise is out of reach); the guard is the estate role's trust. That role must trust BOTH ARNs with the workspace ExternalId (Keyless cloud access page, GET /api/aiops/{tid}/platform-identity) and allow sts:TagSession — the datasource Test runs on the control plane, investigations on the runners. With AIOPS_KEYLESS_AWS_ANY_ACCOUNT: "true" (the example values) the guided CloudShell setup needs no staff step: it creates aiops-<workspace-slug>-read/-write in the account the customer enters, the read role trusting both roles and the write role the control plane only (a role the customer creates by hand must be named aiops-* too). Three cases still need platform staff, through the workspace's GET/PUT /api/aiops/admin/tenants/<id>/cloud-runtime:

   • the account the platform runs in (<customer-account-id>, where a same-account trust alone lets the platform assume the role): publish only that account's exact read/write pair in aws_targets, with no reader. The shared runner (aiops-<slug>-aiops-runner) stays the reader, and the workspace keeps any-account mode for its other accounts;
   • a workspace with a private-network agent: its reads run on the agent's own AWS role, so publish that role as the reader, never the shared runner, and a prepared pair per account;
   • a workspace staff already set AWS up for: its reader (the runner role or a dedicated reader, never the control-plane role) and its pairs.

   Outside <customer-account-id> a prepared role must be named aiops-* with no path: the platform's IAM reaches no other name, and the API refuses such a pair.

4. For the incident's Probable cause card: CloudWatch alarms on the resources in scope routed to /webhook/cloudwatch-sns/<slug> (OK action on the same topic), the workspace harvest interval above 0 (900 s by default), config_history: true and the alarms' accounts and regions on the AWS read connection, and an AWS Config recorder that records the in-scope types, Lambda functions included — see aiops-inputs-needed.md, "What the Probable cause card needs".

Acceptance of the install = the smoke passing + one keyless connection test passing.

3. Upgrade

build-ecr.yml on main publishes <version>-<suffix>.<buildId> (a fixed build-tag suffix set in that pipeline), <version>, the chart charts/aiops:<version> and the signature. The repositories are IMMUTABLE: the plain <version> tag and the chart are pushed by the FIRST build of a version only, and later builds of the same version skip them with a warning. To ship a rebuild, deploy imageTag=<version>-<suffix>.<buildId>; the chart stays the first build's, so a chart change needs a version bump. Then deploy-customer.yml from customer/<slug> with customer=<slug>, imageTag=<new version> (bootstrapDb=false). Rebase customer/<slug> on main first, or merge the release into it, for example git merge v<new version> (a merge needs no force-push): a run uses the pipeline files of the branch it is queued from, so a stale branch deploys with stale pipeline steps, and an empty chartVersion resolves to the version in that branch's pyproject.toml. Read the Plan stage log before approving: an application upgrade should show "No changes" in every unit, at most output-only changes. The Deploy stage runs a fresh terragrunt run-all apply without asking again, so it applies what it plans at that moment, not the plan you read. The pipeline always takes an RDS manual snapshot first and waits for it; the snapshot is named after the tag being deployed, dots as dashes: aiops-<slug>-pre-<tag>-<buildId>, for example aiops-rehearsal-pre-0-1-108-17490. Schema changes are the app's additive automigrate at boot (app/db.py): the old image runs against the newer schema, so a Helm rollback needs no schema rollback. Expect about a minute of downtime (single replica, Recreate).

When the release notes say the deployer policy changed, send the new zip first: the customer re-runs bootstrap.sh apply with the same ExternalId, and it updates only the permissions.

The deploy log artifact deploy-log-<slug> on every run is the record of "what is live where"; the app itself reports its version on /healthz.

Recovery objectives, what the snapshot does and does not cover, and the maintenance windows: availability.md.

4. Rollback

• Manifests / image: --atomic already rolled the release back if the upgrade failed or the pods never became ready. For a bad release that did come up: as a principal listed in eks_admin_principal_arns (the deploy role trusts only our pipeline), aws eks update-kubeconfig --name aiops-<slug>, helm -n aiops history aiops, helm -n aiops rollback aiops <previous revision> --wait. Or re-run deploy-customer.yml with the previous imageTag AND the previous chartVersion (after the release merge an empty chartVersion resolves to the new chart). This is cleaner: it snapshots again and re-verifies the signature. Helm records it as a new upgrade revision, not as a rollback, and its snapshot is named after the version being deployed, so it holds the newer version's database. Rehearsed on 7 October 2026: 0.1.105 → 0.1.108 → 0.1.105 → 0.1.108, each run about 7 minutes from approval, logins and workspaces intact. The release also carries the secret-store keys: once a key rotation has deployed, never helm rollback to a revision older than the last rotation deploy, and re-run a rotation deploy that --atomic rolled back (§12.1, rollback rule).
• Data: only if the new version wrote something the old one cannot read (automigrate is additive, so this is the exception). RDS restore creates a NEW instance, and the settings terraform gave the original do not follow it, so name them: aws rds restore-db-instance-from-db-snapshot --db-instance-identifier aiops-<slug>-pg-restored --db-snapshot-identifier <snapshot from the deploy log> --db-subnet-group-name <same> --vpc-security-group-ids <same> --db-parameter-group-name <same> --multi-az --enable-iam-database-authentication --deletion-protection --backup-retention-period 7 --copy-tags-to-snapshot --manage-master-user-password (--no-multi-az when rds_multi_az = false), and wait until it is available. The deploy always reads the endpoint of the instance named aiops-<slug>-pg and passes it as --set-string externalPostgresql.host, over the values file — so swap the names rather than editing the host: aws rds modify-db-instance --db-instance-identifier aiops-<slug>-pg --new-db-instance-identifier aiops-<slug>-pg-old --apply-immediately, wait until it is available under the new name, then the same call renames aiops-<slug>-pg-restored to aiops-<slug>-pg.
  • Login: the restored instance has a new resource id (DbiResourceId), and rds-db:connect is authorised against that id. The template grants user aiops on any instance of the account (iam unit), so the control plane logs in to the restored one unchanged. An install rendered before that grant pins the original id and is locked out after the swap (PAM authentication failed): bring its iam/terragrunt.hcl up to the template's and apply unit=iam (infra-terragrunt.yml) before the swap.
  • Restart without terraform: as a principal listed in eks_admin_principal_arns, helm -n aiops rollback aiops <previous revision> --wait (the bad release) or kubectl -n aiops rollout restart deployment/aiops (same release; otherwise pooled connections stay on the old instance). Not a deploy run: it applies every terraform unit first, and terraform still tracks the ORIGINAL instance by its resource id (the AWS provider's state id), so its plan renames aiops-<slug>-pg-old back to aiops-<slug>-pg and collides with the restored one.
  • Before the next deploy: reconcile the rds unit's state by hand — remove the original instance from it and import the restored one at the same address — then delete aiops-<slug>-pg-old (deletion protection off) once sure. No pipeline action does this yet, and the sequence has not been drilled.
• In their account today: the cached images (the pull-through-cache repository keeps serving them), the Helm release history, the RDS snapshots. The chart is pulled from OUR registry (helm pull oci://…/charts/aiops --version …, their account is granted pull). The Terraform state lives ONLY in our bucket — there is no copy in their account; the break-glass pack of brief §3.6 (state copy, module mirror, values) is not built.

5. Teardown

1. Agree the date with the customer, and whether they want the final database snapshot (it is kept either way; they delete it).

2. Rebase customer/<slug> on main, or merge main into it (a branch cut before destroy-customer.yml existed cannot queue it), then queue destroy-customer.yml from that branch with customer=<slug> and confirm=<slug>. Stage Plan cannot be unticked under "Stages to run"; it stops before any AWS call unless both match, the slug is a valid one and env.hcl names a customer deploy role (the sandbox env has none, so it can never be the target), then shows terragrunt run-all plan -destroy. Approve on aiops-<slug>: the approval exists only once someone has added it to the Environment (§2, step 4). Stage Destroy runs the same guard again, then, each phase with fresh credentials and a fresh AssumeRole:

   • A: helm uninstall (waits until the Ingress finalizer lets go, i.e. the ALB is gone), deletes namespaces aiops and aiops-agents, waits until no PersistentVolume is left (the EBS volume is deleted with it), switches RDS deletion protection off, and deletes the deploy's aiops-<slug>-pre-* snapshots. The Kubernetes part runs only while the cluster is ACTIVE and still holds the deploy role's access entry; a re-run after a failed cluster delete skips it with a warning;
   • B: terragrunt run-all destroy, retried once after 120 s in a task of its own (ENIs of terminated Auto Mode nodes linger); RDS writes terraform's final snapshot aiops-<slug>-final-… on the way out;
   • C: a read-only leftover check that fails the run while the cluster, the RDS instance, the VPC, an EBS volume of the platform or an Elastic IP tagged Customer=<slug> is left; only then is our state prefix envs/<slug>/ removed. The run then prints what is kept and the follow-ups.

   A failed or timed-out run is re-run as is; the state stays until C is green.

3. The vendor, on main: remove the account id from platform_image_pull_account_ids and apply ecr and ecr-charts; delete the variable group and the Environment; keep or delete branch customer/<slug>.

4. Customer: bash deploy/customers/bootstrap.sh delete in CloudShell opened in the install region (with the same AIOPS_ACCOUNT_ID, plus AIOPS_HOST to delete the certificate it requested, once no load balancer uses it). It refuses while an aiops-* cluster, an aiops-*-pg instance or an aiops-*-vpc VPC exists in ANY enabled region (a region the organisation's SCPs close is skipped and named): a teardown that stopped half-way needs the role to finish, and a role recreated later would not get back into the cluster (EKS access entries do not follow a new role of the same name). Then they remove the DNS records and, once sure, the final snapshot.

Kept after the teardown: terraform's final DB snapshot, and the EKS secrets KMS key until its scheduled deletion date. What they cost is in §6.

6. Costs and timings

Times: the 6 October 2026 rehearsal on build 0.1.105, as in §0 (02-install.md); no cost has been measured yet. A list-price estimate for these defaults (about USD 725–755 a month before model usage) is in Expected monthly cost.

What the running stack is made of: the EKS control plane, Auto Mode nodes and the Auto Mode fee, one NAT gateway, RDS db.t4g.medium Multi-AZ with 20 GB gp3 and 7 days of backups, one ALB, VPC endpoints (an S3 gateway endpoint, free; interface endpoints for ECR api, ECR dkr, Secrets Manager, STS and the Bedrock runtime, each in two AZs at about USD 7–10 per AZ a month plus a per-GB charge), public IPv4 addresses, CloudWatch logs and alarms, and model usage per investigation.

Availability knobs (env.hcl, applied by the next deploy): rds_multi_az (default true — a synchronous standby in the second AZ, about twice the instance cost; the sandbox runs false), rds_max_allocated_storage (default 100 GiB — the autoscaling ceiling over the 20 GB start, billed only as it grows), single_nat_gateway (default true — one NAT for both AZs; false adds a second NAT gateway for AZ-resilient egress), enable_core_endpoints (default true — the S3, ECR, Secrets Manager and STS endpoints, about USD 60–80 a month for the four interface endpoints; false sends that traffic through the NAT) and enable_bedrock_endpoint. The RDS unit also creates two CloudWatch alarms — free storage below 2 GiB, connections above 80 % of the class limit — notifying the customer's topics in alarm_sns_topic_arns and, when alarm_email is set, the topic the unit creates, aiops-<slug>-alarms, with that address subscribed (SNS emails it a confirmation link that must be clicked once; at this volume the topic costs nothing). Both empty (the default): the alarms exist, nobody is paged.

Cost guardrails (enable_cost_guardrails, default false): a monthly AWS Budget of cost_budget_monthly_usd that mails alarm_email at 80 % and 100 % of the actual spend and at 100 % of the forecast, and a daily Cost Anomaly Detection summary of anomalies worth USD 100 or more. Only in an account that owns its billing: in a member account of an AWS Organization, budgets and anomaly detection belong to the payer, the APIs answer AccessDenied and the deploy fails — the payer sets them there. An account holds one AWS-services anomaly monitor; when it has one already, put its ARN in cost_anomaly_monitor_arn and the subscription uses it.

7. Not covered

• Option B, the customer running the install themselves (CloudShell hands over to a CodeBuild job in their account, since CloudShell ends idle sessions): on request; it moves the Terraform state, secrets and upgrades to their side.
• GitHub Actions in the customer's GitHub: on request (needs a GitHub OIDC trust on our infra role and the install extracted from the pipeline).
• Image mirror / tar export (each release pushed into their registry): on request; the pull-through cache is the default.
• ACR, Azure / GCP accounts: the trust chain (workload identity / Entra app), the registry path (ACR's cache cannot pull from a private ECR) and the signing-key location differ.
• The permissions boundary on aiops-deployer (§1.4).
• Private EKS endpoint (needs a self-hosted agent in their VPC), an internal ALB, ECR cross-account replication (the no-NAT fallback), in-cluster admission verification (their Kyverno/Ratify; we hand over aiops-release.pub on request once signing is on).
• Air gap = profile 2 (Zarf package, --verify always), design only; its real prerequisite is an LLM endpoint inside the boundary.
• Licence enforcement: no key issued yet, so every install runs unlicensed — /ops → Platform shows "Unlicensed — no enforcement" and nothing is paused. The slot is wired (§11).
• Image signing: off until the signing key exists; both pipelines then warn that images are unsigned/unverified (§9).
• Digest pinning in Helm: the pipeline verifies and records the digest but the chart has no image.digest value, so the pod pulls by tag (one template line closes it). Reproducible image builds: kubectl and helm are pinned, but the python:3.12-slim base tag and the Debian security updates applied at build time float, so two builds of one commit can differ — pin the base digest before a customer asks us to reproduce a signed digest.

8. Proof

• Dry run, 2026-09-28, in our trial account in ap-southeast-1, which is not a fresh account (service-linked roles, quotas and Bedrock state already existed there, beside other workloads), with the earlier by-hand steps: install, /healthz gate, HTTPS through the ALB, an upgrade through the pipeline and a helm rollback all passed. It took four pipeline runs, each fixing a real defect now in this lane (the trust statement split, the ECR pull-through service-linked role, the db bootstrap without kubectl attach, no $ in the bootstrap SQL). The smoke passed on Nova Pro; AWS refused Anthropic models for that account. The teardown was done by hand; destroy-customer.yml is meant to replace it, and its first run is the rehearsal below. What only a brand-new account shows (first-use service-linked roles, a small vCPU quota, Bedrock with nothing on file) is proven only by the customer's own first install.
• Rehearsal of this runbook, 6 October 2026, build 0.1.105: a fresh install into a separate AWS account under the slug rehearsal — bootstrap plan and apply, deploy and the real-model smoke, timed in §0 and §6 (02-install.md). The certificate was already issued and the pull grant already existed, so neither was timed. The same install took the §3 upgrade and the §4 rollback on 7 October (0.1.105 → 0.1.108 → 0.1.105 → 0.1.108) and a point-in-time restore drill on 8 October (record). No record yet of bootstrap delete or of a teardown through destroy-customer.yml with its leftover check, so §0 row T and every cost in §6 stay unmeasured.

9. One-time setup on our side (not per customer)

• The CI role policies are applied by hand, not by terraform: infrastructure/iac/oidc-role/infra-role-policy.json on aiops-demo-ado-infra (its AssumeCustomerDeployRole statement allows sts:AssumeRole and sts:SetSourceIdentity on role/aiops-deployer only) and role-policy.json on the build role.
• Signing (optional until the key exists): create the KMS key once — aws kms create-key --key-spec ECC_NIST_P256 --key-usage SIGN_VERIFY + aws kms create-alias --alias-name alias/aiops-release --target-key-id <id>; re-apply role-policy.json (CosignReleaseKey) to the build role; export cosign public-key --key awskms:///alias/aiops-release > deploy/customers/aiops-release.pub; set COSIGN_KMS_ALIAS: alias/aiops-release in BOTH build-ecr.yml and deploy-customer.yml. Where that key lives follows the registry-host decision (decision log §6.4).

10. Sandbox notes that affect this path

• Demo sync window. On the demo platform (ArgoCD, not this pipeline) the AppProject aiops (infrastructure/gitops/apps/aiops/project.yaml) denies automated syncs of the aiops and aiops-cloud-proof-primary-reader Applications 09:00–10:30 UTC Mon–Fri — the customer's daily sync, where a push-triggered Recreate rollout cost 11 minutes on 01-Oct. image-updater still records the new tag during the window; the rollout runs when it closes. A hotfix inside the window is a manual Sync in ArgoCD (allowed by the window). Commits that touch only infrastructure/ do not build an image and roll nothing. The customer path is untouched: deploy-customer.yml is run by hand, so its timing is yours.
• The sandbox db-init Job (infrastructure/gitops/apps/db-init/, hardcoded RDS host, rode ArgoCD) was removed on 06-Oct-2026: the databases and roles exist, and the Job had been failing on every sync since the RDS-managed master password (manage_master_user_password) rotated away from the copy in the aiops-db Secret. A cold rebuild of the sandbox needs it back: restore the folder from git (last present at 6b294eb3) after copying the current RDS-managed master password (rds!db-… secret) into RDS_MASTER_PASSWORD in Secrets Manager aiops-demo/database. The customer path never used this Job — deploy-customer.yml (bootstrapDb) creates the role itself.
• infrastructure/gitops/values/aiops.yaml still pins build 0.1.0-<suffix>.15787 as the bootstrap tag (image-updater writes back to ArgoCD, not git); irrelevant to customers, but a cold ArgoCD bootstrap of the sandbox starts ~80 releases behind.

11. Licence

An offline signed file, no phone-home (licence.md): the pod verifies the JWT in AIOPS_LICENSE against licence.publicKey at boot and every scheduler tick. No key exists yet (alias/aiops-licence waits on the vendor-of-record decision, decision log §6): every install runs unlicensed = no enforcement. The slot is wired, so that changes with one variable.

• Install: AIOPS_LICENSE is a secret variable of aiops-<slug>-deploy (mint writes it empty; an older group gets it from ado <slug> --update-group); the deploy passes it as --set-string licence.value. The public key is licence.publicKey in the values file on customer/<slug>.
• Renew: the new token as AIOPS_LICENSE in ~/.aiops/customers/<slug>.env, ado <slug> --update-group, then deploy-customer.yml with the current imageTag — a rollout, no endpoint. The file is the source of truth (Step 4): a token pasted only in the ADO Library goes back to the file's value at the next --update-group.
• Issue: scripts/issue-licence.py (licence.md). kms:Sign on the alias belongs to the role of an approval-gated issue job only, never to the build role, which every push to main and the power schedule assume. That job and its role are not built yet.

12. Rotating secrets

The secrets file ~/.aiops/customers/<slug>.env stays the source of truth (Step 4): change a value there, run ado <slug> --update-group, then deploy-customer.yml from customer/<slug> (rebased on main) with the live imageTag. A changed Secret restarts the control plane through its checksum, and the provisioner recreates every runner whose environment changed.

12.1 The tenant secret-store key

AIOPS_SECRET_KEY encrypts every credential the platform stores. A rotation keeps the old key next to the new one as AIOPS_SECRET_KEY_PREVIOUS (it still decrypts, never encrypts) until the control plane has re-keyed every stored secret, which it does at boot: one compare-and-swap per row, a secret.rekeyed audit row per secret in its workspace's audit log, one secrets.rekey_completed summary in the platform audit log. A secret no configured key opens is never touched, and an edit of that connection answers 409 instead of overwriting it. The re-key needs a build with this support, deployed once before the first rotation: that boot stamps every stored secret with the current key.

1. Precheck. AIOPS_SECRET_KEY_PREVIOUS is empty in the secrets file (no rotation open). Copy the secrets file, at least the current key, to the password vault. Take the census on the install — an EKS admin principal (the customer's, §4) runs it and sends back the JSON, which holds counts, a key fingerprint and opaque secret ids, never a secret:

   kubectl -n aiops exec deploy/aiops -- python -m app.aiops.secrets rekey --dry-run > census.json
   

   It must show "remaining": 0 and "unreadable": []; note readable. Each deploy-customer.yml run takes an RDS snapshot before the helm upgrade.

2. Rotate. python3 scripts/customer-onboard.py rotate-key <slug> --census census.json — it refuses without a clean census of the current key, then makes the current key AIOPS_SECRET_KEY_PREVIOUS and mints a new AIOPS_SECRET_KEY. Copy the file to the vault again: it holds the only copy of the new key. Then ado <slug> --update-group (a human with ADO_PAT).

3. Deploy with both keys. Rebase or merge customer/<slug> with main (the mapping of the previous key lives in the branch's deploy-customer.yml), then queue deploy-customer.yml with customer=<slug> and the live imageTag; the customer approves. The control plane restarts with both keys and re-keys at boot; db-mode runners restart with both keys.

4. Check. Ops › Audit log, the newest secrets.rekey_completed: key_id is the new fingerprint rotate-key printed, remaining 0, unreadable [], readable the census count (give or take connections added or deleted since). Each workspace's audit log shows secret.rekeyed per stored secret. Test the connections.

5. Retire. Save that event as JSON (copy it from Ops › Audit log, or GET /api/aiops/admin/audit?action=secrets.rekey_completed&limit=1 with a superadmin token), then python3 scripts/customer-onboard.py retire-key <slug> --rekey-report event.json. It refuses unless the event is for the new key with remaining 0 and unreadable [], then moves the previous key to AIOPS_SECRET_KEY_RETIRED_<YYYYMMDD>, a line that never goes to Azure DevOps. ado <slug> --update-group, deploy again, Test again.

Stop rule. unreadable not empty or remaining above 0: do not retire. The usual cause is a previous key that never reached the pod (a customer branch not rebased, or update-group skipped): the secrets stay untouched and the connections fail closed. Fix the plumbing and deploy again — the boot re-key resumes — or roll back: swap the two values in the secrets file (the old key back as AIOPS_SECRET_KEY, the new one as AIOPS_SECRET_KEY_PREVIOUS), update-group and deploy; the boot re-key moves whatever was written under the new key back to the old one, and step 5 closes the rollback like any rotation.

Rollback rule. The release carries the keys (they ride --set-string into the chart Secret), so rolling it back can undo a rotation step. A helm rollback to a revision from before step 3, or --atomic undoing a step-3 run that failed after its pod had booted and re-keyed, brings the old key back alone. Every secret already re-keyed then fails closed (connections down, edits 409) until the next pipeline deploy, and a credential saved meanwhile is written under the old key: once that key is retired, only a deploy carrying the retired line as AIOPS_SECRET_KEY_PREVIOUS opens it again (as in the DR rule), and step 5 then closes that like any rotation. From step 3 on:

• roll back only by re-running deploy-customer.yml, which renders the current keys from the group (to undo the rotation itself, swap the keys as the stop rule says); never helm rollback to a revision older than the last rotation deploy (step 3 or step 5);
• a step-3 run that failed is re-run, and must succeed, before step 4;
• retire only on the newest secrets.rekey_completed, written after the last successful deploy run started: an older event does not count what was saved under the old key since.

DR rule. Keep the AIOPS_SECRET_KEY_RETIRED_<YYYYMMDD> line until the oldest snapshot and point-in-time-restore window from before the rotation have expired: a restore of such a backup needs that key — deploy the restored install with it as AIOPS_SECRET_KEY_PREVIOUS and the boot re-key moves its secrets onto the current key.

db-mode runners restart at each deploy of a rotation (their environment changes); api-mode runners never receive either key.

Compose. The same steps with the .env file as the source of truth:

docker compose exec aiops python -m app.aiops.secrets rekey --dry-run  # census: remaining 0, unreadable []
# .env: AIOPS_TENANT_SECRET_KEY_PREVIOUS=<the old key>, AIOPS_TENANT_SECRET_KEY=<a new key>
docker compose up -d                                                    # restarts with both keys, re-keys at boot
docker compose exec aiops python -m app.aiops.secrets rekey --dry-run  # remaining 0, unreadable [] on the new key
# .env: clear AIOPS_TENANT_SECRET_KEY_PREVIOUS (keep the old key in the vault), then
docker compose up -d

Runner containers the control plane provisioned itself (AIOPS_RUNNER_PROVISIONER=docker) keep the environment they were created with. After each up -d, stop them with their drain window and remove them; the control plane provisions them again with the current keys:

ids=$(docker ps -aq --filter label=aiops.managed=1); [ -z "$ids" ] || { docker stop -t 120 $ids; docker rm -f $ids; }

Next steps

• Next: Licence
• All Install and operate the platform pages
• Documentation home