# Customer deploy runbook — a fresh AWS account in 8 steps

> The managed install into one dedicated AWS account in eight steps, with upgrade, rollback, teardown, costs, licence and secret rotation.

_Reference · Updated 2026-10-19 · Describes release 0.1.164_

Managed BYOC, profile 1 (decision brief 2026-09-18; decision log §6.4): the customer owns one dedicated AWS account and one IAM role our pipeline may assume. Our Azure Pipelines create the stack in that account through that role and `helm upgrade --install` our OCI chart. Nothing of ours runs 24x7 in their account beyond the platform itself: no ArgoCD, no licence server, no Kyverno.

Files this runbook drives:

| Piece | Path |
|---|---|
| Customer script (they run it) | `deploy/customers/bootstrap.sh` + `aiops-deployer-trust.json`, `aiops-deployer-policy.json` beside it |
| Our onboarding script | `scripts/customer-onboard.py` (`mint`, `render`, `ado`; `rotate-key`, `retire-key` in §12.1) |
| Terraform env template | `infrastructure/iac/envs/_customer-template/` (+ its README) |
| Provider role chain, state region | `infrastructure/iac/terragrunt.hcl` (`deploy_role_arn`, `deploy_external_id`, `state_region`) |
| Our-side pull grant | `infrastructure/iac/envs/<host-env>/env.hcl` (our sandbox host environment) `platform_image_pull_account_ids`, units `ecr`, `ecr-charts` |
| Build: images + chart + signature | `.azuredevops/build-ecr.yml` |
| Install / upgrade per customer | `.azuredevops/deploy-customer.yml`, `.azuredevops/steps/customer-env.sh` |
| Teardown per customer | `.azuredevops/destroy-customer.yml` |
| Values | `deploy/values-customer.example.yaml` -> `deploy/customers/<slug>/values.yaml` (rendered) |
| Post-install smoke | `scripts/customer-smoke.sh` |
| Chart | `deploy/aiops-helm` (pushed as `oci://<our-registry>/charts/aiops:<version>`) |

`<slug>` is one string used everywhere: env dir, values dir, branch `customer/<slug>`, ADO Environment `aiops-<slug>`, variable group `aiops-<slug>-deploy`. The per-customer files carry the customer's account id and host name, so they live on branch `customer/<slug>` and never on main.

## 0. The whole install {#0-the-whole-install}

The host owner is whoever holds the DNS zone of the platform's host name: us for a test host of ours, the customer for their own domain.

| # | Customer | Vendor | Time (6 October 2026 rehearsal) |
|---|---|---|---|
| 1 | — | Mint the ExternalId and the app secrets (`customer-onboard.py mint`); send the zip it names, and the ExternalId in a separate message | not measured |
| 2 | In CloudShell opened in the install region: `bootstrap.sh plan`, then `apply`; send back the JSON it saves | — | plan 14 s, apply 19 s |
| 3 | Host owner: add the certificate validation CNAME the bootstrap printed; wait for `ISSUED` | (host owner, for our test host) | not measured: the certificate was already `ISSUED` and was reused |
| 4 | — | `render` on branch `customer/<slug>`, `ado` + an approver on the Environment, the pull grant on main | not measured; the pull grant already existed |
| 5 | Reachable during the booked window | `deploy-customer.yml` with `bootstrapDb=true`; approve | about 27 min from approval to the passing HTTPS health gate, pipeline waits included; inside it, infrastructure apply 18 min 21 s and Helm install 2 min 17 s |
| 6 | Host owner: CNAME the host name to the load balancer the deploy printed | (host owner, for our test host) | not measured |
| 7 | — | Model price map if needed, then `scripts/customer-smoke.sh` | real-model smoke 34 s |
| 8 | Log in, change the admin password, connect the IdP, the first read role and the alert sources | Publish the workspace's runtime reader under /ops | not measured |
| T | `bootstrap.sh delete`, the DNS records, the final snapshot once sure | `destroy-customer.yml`; pull grant, variable group, Environment | not measured |

Times are those observed in the 6 October 2026 rehearsal on build 0.1.105 ([02-install.md](/help/ops-install.md#timings-and-scope-of-proof)): measurements, not an SLA. The step 5 phases overlap and sit inside its interval, so do not add them up.

## 1. Before you start {#1-before-you-start}

### 1.1 The account {#1-1-the-account}

- A **dedicated** AWS account, or a new member account in the customer's Organization (§1.4 explains why dedicated).
- Ask about **SCPs, permission boundaries and mandatory tags** before step 2. The bootstrap tags what it creates with `Project` and `ManagedBy` only; the stack's tags come from `env.hcl` (`tags`).
- Step 2 runs as an account admin: it creates an IAM role, may request an ACM certificate, and reads EC2, Service Quotas and Bedrock.

### 1.2 Region {#1-2-region}

- EKS Auto Mode and Bedrock available (or the customer's own OpenAI-compatible endpoint), and two availability zones EKS supports. The bootstrap picks two and skips the AZ ids EKS cannot use (`use1-az3`, `usw1-az2`, `cac1-az3`).
- Images cross regions through the ECR-to-ECR pull-through cache (supported across Regions and accounts); our registry stays in `ap-southeast-1`.
- Terraform state stays in our bucket: `state_region` in `env.hcl` names the bucket's region, so the customer region may differ from it.
- A fresh account or region can have a small On-Demand vCPU quota; the bootstrap warns below 8.
- Proven so far: `ap-southeast-1` (§8). Any other region is proven by its first install.

### 1.3 Model {#1-3-model}

- **Claude** through a Bedrock inference profile of `AIOPS_ANTHROPIC_MODEL` (default `anthropic.claude-sonnet-5`). The bootstrap picks the geographic profile (`apac.`, `eu.`, `us.` …). A `global.` profile routes requests worldwide, a data-residency question the customer answers: it is picked only with `AIOPS_ALLOW_GLOBAL_ROUTING=true`, and when the region offers Claude only through `global.`, the bootstrap says so and recommends Nova Pro until they opt in. In THIS account:
  1. Anthropic's use-case form is on file — submitted in this account's Bedrock console, or inherited from the Organization's management account.
  2. The model agreement exists — created by one first invocation from the console Playground by an admin allowed `aws-marketplace:Subscribe`, or by `aws bedrock create-foundation-model-agreement` (AWS CLI 2.27.42 or later). The bootstrap prints the two commands (list the offer, then create the agreement) and never runs them: accepting the model's terms is the customer's decision. A `PENDING` agreement is waited for, never created twice.
  3. The rest of Bedrock's availability answer allows it: the account is entitled to the model (AWS can withhold Anthropic models from an account), IAM and SCPs authorise it, and the region serves it. The bootstrap recommends Claude only when all four fields allow it, and otherwise names the field that does not in `model.anthropic_next_step`.

  The platform's Pod Identity roles hold `bedrock:InvokeModel*` only and cannot create the agreement. While Claude is not available to the account, every investigation fails on the runner with `… is not available for this account`.
- **Nova Pro** (`bedrock/<geo>.amazon.nova-pro-v1:0`) otherwise: the bootstrap's `model.recommended` picks it automatically when Claude is not available, and `render` copies it into `AIOPS_MODEL`.
- **Their OpenAI-compatible endpoint**: `render --openai-endpoint`, then set `AIOPS_MODEL=openai/<model>`, `AIOPS_API_BASE` and `AIOPS_API_KEY` in the secrets file before `ado`.

Check before booking the install date: on 2026-09-28 AWS refused Anthropic models for our test account even after the form ("contact AWS Sales").

### 1.4 What the deploy role can do {#1-4-what-the-deploy-role-can-do}

`aiops-deployer` holds one inline policy, `aiops-deployer-policy.json`: what the `_customer-template` units create, unit by unit (checked against the module sources: vpc 6.0.1, security-group 5.3.0, eks 21.0.6, rds 6.10.0, vpc-endpoints 6.0.1, and the local `ecr-cache` / `iam` / `alb-access` / `cost-guardrails` tf).

| Statement | Needed by |
|---|---|
| `TheStackItself` — `ec2:*`, `eks:*`, `ecr:*`, `rds:*`, `elasticloadbalancing:*`, `logs:*`, `cloudwatch:DescribeAlarms`, KMS key admin, `sts:GetCallerIdentity`, `tag:GetResources` | `network`, `rds-sg`, `alb-access`, `endpoints` (EC2); `eks` (cluster, access entries, log group, KMS key + alias for secrets encryption); `iam` (Pod Identity associations); `ecr-cache` (pull-through rule + cached repository); `rds` (instance, subnet + parameter group, reading its alarms); the pipelines' own snapshot / describe / kubeconfig / leftover calls |
| `ServiceLinkedRoles` | first EKS cluster, first RDS instance and first ECR pull-through rule in a fresh account (`AWSServiceRoleForAmazonEKS`, `...ForRDS`, `...ForECRPullThroughCache`), created on first use through this role; the bootstrap does not pre-create them |
| `IamForTheStack` on `role/aiops-*`, `policy/aiops-*`, `instance-profile/aiops-*`, `oidc-provider/*` | eks cluster + Auto Mode node roles, the `ClusterEncryption` and Auto Mode tag policies, the IRSA OIDC provider (delete included, for teardown); `iam/` Pod Identity roles; `ecr-cache/` upstream role; `iam:PassRole` for all of them. `aiops-*`, not `aiops-<slug>*`: the eks module reads its CALLER (`data "aws_iam_session_context"` = `iam:GetRole` on `aiops-deployer` itself) |
| `RdsManagedMasterSecret` on `secret:rds!*` | `manage_master_user_password` (RDS creates the secret with the caller's rights), the first-install bootstrap reading it, deletion at teardown |
| `RdsAlarms` on `alarm:aiops-*` | `rds` — the two CloudWatch alarms beside the instance (`logs:*` is CloudWatch Logs, another service): `PutMetricAlarm` plus `TagResource` (the provider's `default_tags` ride the create), `ListTagsForResource` on refresh, `DeleteAlarms` at teardown |
| `AlarmTopic` on `sns:*:*:aiops-*` | `rds` — with `alarm_email` set, the topic `aiops-<slug>-alarms` (created tagged) and its email subscription: create, refresh, delete at teardown |
| `CostGuardrailsBudget` on `budget/aiops-*`, `CostGuardrailsAnomaly` | `cost-guardrails`, only with `enable_cost_guardrails`: the monthly budget `aiops-<slug>-monthly`, the AWS-services anomaly monitor and its subscription (their ARNs carry generated ids, so the `ce:` actions are not name-scoped) |
| `InterfaceEndpointPrivateDns` | `endpoints/` — an interface endpoint with private DNS associates a Route 53 private zone with the VPC |

**Said plainly:** `IamForTheStack` allows `iam:AttachRolePolicy`, `iam:PutRolePolicy` and `iam:CreatePolicyVersion` on `role/aiops-*` and `policy/aiops-*` with no `iam:PolicyARN` condition, and `aiops-deployer` itself matches `role/aiops-*`. The role can therefore attach any policy, `AdministratorAccess` included, to itself: in practice it can make itself administrator of the account. The permissions boundary described in the 18-Sep decision brief (§3.1) is **not built**. Containment today is the dedicated account, plus the fact that only our pipeline can assume the role (our infra role, the ExternalId we minted, and a SourceIdentity naming the operator), so every session and every call lands in the customer's CloudTrail under that name. The boundary is the hardening before any production install, or before a test if the customer asks for it.

The trust (`aiops-deployer-trust.json`) allows our infra CI role only: `sts:AssumeRole` with the ExternalId, a Deny on `sts:AssumeRole` without a SourceIdentity, and separate `sts:SetSourceIdentity` and `sts:TagSession` statements. The ExternalId key exists only in AssumeRole's request context — on a statement that also lists SetSourceIdentity or TagSession it never matches, and STS refuses the whole call. Access Analyzer reports no findings on this document. Our side of the chain is the `AssumeCustomerDeployRole` statement in `infrastructure/iac/oidc-role/infra-role-policy.json`.

### 1.5 Host name and certificate {#1-5-host-name-and-certificate}

- **A test**: our test host name, so the customer has no DNS work. The bootstrap requests the certificate in their account (`AIOPS_HOST`), and we add both CNAMEs (validation, then the load balancer) in our zone.
- **Production**: their host name. They add both CNAMEs, or bring an ISSUED certificate of their own (`AIOPS_CERT_ARN`).
- DNS-only records: a proxying CDN in flexible-TLS mode loops on the ALB's `ssl-redirect`. No external-dns, no cert-manager.

### 1.6 Network {#1-6-network}

- The stack keeps a NAT gateway: the first pull through the cache rule, webhooks and tickets need it. A fully private VPC switches image delivery to ECR replication, which is not built.
- A path to their estate (peering / TGW / PrivateLink), or the private-network agent (`deploy/aiops-agent-helm`) in their VPCs. `render --estate-cidr` opens the runner's egress to their private estate.

## 2. Install, step by step {#2-install-step-by-step}

### Step 1 — Vendor: mint {#step-1-vendor-mint}

```bash
python3 scripts/customer-onboard.py mint <slug>
```

Writes `~/.aiops/customers/<slug>.env` (mode 0600) once and never overwrites it: `AIOPS_DEPLOY_EXTERNAL_ID` (a UUID: the confused-deputy guard, minted by us per customer), `SECRET_KEY`, `ADMIN_PASSWORD`, `AIOPS_SECRET_KEY` (the Fernet key of the tenant secret store: the file variable reaches the pod as `AIOPS_TENANT_SECRET_KEY`; losing it loses every stored datasource credential, so back the file up like a key; rotating it is §12.1) and empty `AIOPS_SECRET_KEY_PREVIOUS`, `AIOPS_API_KEY`, `AIOPS_MODEL`, `AIOPS_API_BASE`. It prints the hand-off: the zip command (`git archive` of `bootstrap.sh` and the two JSON files, run from a clean checkout of main) and the `sha256sum` of the same three files, the ExternalId, and the customer's command line. Send the zip with the sums, and the ExternalId in a separate message. A second `mint` prints the same hand-off again.

### Step 2 — Customer: bootstrap {#step-2-customer-bootstrap}

In AWS CloudShell opened in the install region, as an account admin:

```bash
unzip aiops-bootstrap.zip
export AIOPS_ACCOUNT_ID=111122223333 AIOPS_EXTERNAL_ID=<the ExternalId from the vendor>
export AIOPS_HOST=aiops.example.com      # optional: request the certificate here
bash deploy/customers/bootstrap.sh plan  # read-only
bash deploy/customers/bootstrap.sh apply
```

| Input | Meaning |
|---|---|
| `AIOPS_ACCOUNT_ID` | this account; the first AWS call checks it and stops on any other |
| `AIOPS_EXTERNAL_ID` | the UUID from step 1 |
| `AIOPS_HOST` | optional: the platform host name; `apply` reuses an ACM certificate for exactly that name or requests one (DNS validation) |
| `AIOPS_CERT_ARN` | optional: an existing certificate for `AIOPS_HOST` in this account and region |
| `AIOPS_REGION` | default: the region CloudShell was opened in (`AWS_REGION`) |
| `AIOPS_ADMIN_ROLE_ARN` | the cluster's break-glass admin when the caller cannot be one (root, a federated session); default: the caller's IAM user, or its role for an SSO / assumed-role session |
| `AIOPS_VENDOR_ROLE_ARN` | default `arn:aws:iam::<vendor-account-id>:role/aiops-demo-ado-infra`, our pipeline role |
| `AIOPS_ANTHROPIC_MODEL` | default `anthropic.claude-sonnet-5`: the Bedrock model id whose inference profile the bootstrap looks for (§1.3) |
| `AIOPS_ALLOW_GLOBAL_ROUTING` | `true` lets the bootstrap pick a `global.` profile, which may serve requests from any region; default: geographic profiles only |

**An account with only its root user.** `plan` stops when the caller is the account root (or a federated session): EKS access entries take an IAM user or role, and root cannot assume a role, so root cannot be the cluster's break-glass admin. Create a role for the people who will use it and name it before `plan`:

```bash
cat > break-glass.yaml <<'EOF'
Resources:
  BreakGlass:
    Type: AWS::IAM::Role
    Properties:
      RoleName: platform-break-glass
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: sts:AssumeRole
      Policies:
        - PolicyName: eks-describe-cluster
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: eks:DescribeCluster
                Resource: "*"
EOF
aws cloudformation deploy --stack-name platform-break-glass --template-file break-glass.yaml \
  --capabilities CAPABILITY_NAMED_IAM --no-fail-on-empty-changeset
export AIOPS_ADMIN_ROLE_ARN=arn:aws:iam::$AIOPS_ACCOUNT_ID:role/platform-break-glass
```

The deploy makes that ARN a cluster admin through an EKS access entry (`render` copies the bootstrap JSON's `admin_principal_arn` into `eks_admin_principal_arns`); the role's own policy only lets `aws eks update-kubeconfig` read the cluster. Its trust names the account, so an IAM user or SSO role of yours that may call `sts:AssumeRole` on it can use it: give at least one person such an identity, since root cannot. Never name it `aiops-*`: the platform's control-plane and runner roles may assume any role so named, and in this account a trust that names the account is all they need.

`apply` creates or updates the role `aiops-deployer` from the two JSON files (trust rendered with our pipeline role and the ExternalId, 1-hour sessions, tagged `ManagedBy=aiops-bootstrap`), and writes only what differs: a second `apply` makes no change. It refuses an existing `aiops-deployer` it did not create and whose trust does not name our pipeline role. `plan` and `apply` also check the availability zones, the vCPU quota and Bedrock access, and print one JSON (saved as `./aiops-bootstrap-<account>.json`) that holds no secret — never the ExternalId:

`account_id`, `region`, `azs`, `deploy_role_arn`, `deploy_role_ready`, `admin_principal_arn` (goes into the cluster's access entries, so someone of theirs can always `kubectl` in), `host`, `certificate_arn`, `certificate_status`, `validation_record`, `model` (`recommended`, `anthropic_agreement`, `use_case_form`, `anthropic_next_step`, `claude_profile`, `nova_profile`), `bedrock_in_region`, `ec2_standard_vcpu_quota`, `existing_vpc_cidrs`, `policy_sha256`.

A read it cannot make (the vCPU quota, Bedrock) is printed as a `note:` line with the AWS error instead of being dropped. They send back the file `apply` wrote. A `plan` file says `deploy_role_ready: false` until the role exists, and `render` refuses it.

### Step 3 — Certificate validation {#step-3-certificate-validation}

The host owner adds the validation CNAME from `validation_record` (DNS only) and waits until ACM shows `ISSUED`; re-running `bootstrap.sh plan` prints the status. The ALB gets no HTTPS listener until the certificate is ISSUED.

### Step 4 — Vendor: render, Azure DevOps, pull grant {#step-4-vendor-render-azure-devops-pull-grant}

```bash
git switch -c customer/<slug>
python3 scripts/customer-onboard.py render <slug> aiops-bootstrap-111122223333.json \
  [--estate-cidr 10.20.0.0/16] [--openai-endpoint] [--alarm-email aiops-oncall@example.com]
git add infrastructure/iac/envs/<slug> deploy/customers/<slug>
git commit -m "chore(deploy): add <slug>" && git push -u origin customer/<slug>
ADO_PAT=<a PAT of yours> python3 scripts/customer-onboard.py ado <slug>
```

- `render` copies `_customer-template` to `infrastructure/iac/envs/<slug>/` and fills `env.hcl` (region, account, our state bucket, deploy role, name prefix and cluster name `aiops-<slug>`, the `Customer` tag, the AZs, `eks_admin_principal_arns`, `enable_bedrock_endpoint = false` for `--openai-endpoint` or a region without Bedrock, and `alarm_email` — the RDS alarms' named recipient, a role mailbox since it is committed — from `--alarm-email`; the address confirms the subscription mail AWS sends after the deploy). It fills `deploy/customers/<slug>/values.yaml` from the example (image repository, host, certificate, public URL, the keyless trust anchor role and account, region, runner role, the VPC and estate CIDRs; without `--estate-cidr` the estate rule repeats the VPC CIDR) and sets `AIOPS_MODEL` in the secrets file from `model.recommended`. It runs the deploy's own placeholder guard and refuses: off branch `customer/<slug>`, a `plan` JSON, a JSON without host or certificate, any value not of its exact shape (terragrunt evaluates `env.hcl` with our pipeline credentials, so the customer's file is checked field by field), no usable model without `--openai-endpoint`, and files it rendered before unless `--force` (a re-render drops hand edits such as a changed `vpc_cidr`; read `git diff` before committing). It warns when the VPC CIDR overlaps one of theirs, the certificate is not ISSUED yet, or their deploy role carries another version of the policy than this commit. The RDS endpoint is not filled in: the deploy reads it from the instance. The ExternalId never lands in git: `deploy_external_id` reads it with `get_env()`.
- `ado` (a human runs it: it grants) registers `deploy-customer.yml` and `destroy-customer.yml` if missing, creates Environment `aiops-<slug>`, creates variable group `aiops-<slug>-deploy` from the secrets file, and authorises both pipelines on the group, the Environment and the `aws-oidc-infra` service connection. It reads `ADO_PAT` from the environment only. Re-running it creates nothing new and leaves an existing group as it is. The secrets file is the source of truth: to change a value, edit the file and run `ado <slug> --update-group`, which rewrites every value of the group from the file (a value changed only in ADO is lost).

  | Variable | Secret | Meaning |
  |---|---|---|
  | `AIOPS_DEPLOY_EXTERNAL_ID` | yes | the ExternalId in their trust policy |
  | `SECRET_KEY` | yes | `secrets.AIOPS_SECRET_KEY` — signs only the 10-minute SSO sign-in state and the 24-hour SSO logout hint. Sessions are stored hashed in the database, so rotating it signs nobody out; an SSO sign-in in progress restarts |
  | `ADMIN_PASSWORD` | yes | `secrets.AIOPS_ADMIN_PASSWORD` — bootstrap superadmin |
  | `AIOPS_SECRET_KEY` | yes | `secrets.AIOPS_TENANT_SECRET_KEY` — Fernet key of the tenant secret store |
  | `AIOPS_SECRET_KEY_PREVIOUS` | yes | `secrets.AIOPS_TENANT_SECRET_KEY_PREVIOUS` — the store key being rotated out (§12.1); empty otherwise |
  | `AIOPS_API_KEY` | yes | LLM gateway key; empty on Bedrock |
  | `AIOPS_LICENSE` | yes | `licence.value` — the offline licence JWT (§11); empty = unlicensed |
  | `AIOPS_MODEL` | no | `bedrock/<inference-profile-id>` or `openai/<model>` |
  | `AIOPS_API_BASE` | no | the OpenAI-compatible base; empty on Bedrock |

  The deploy refuses an empty `AIOPS_MODEL` (that would be the mock LLM), `SECRET_KEY`, `ADMIN_PASSWORD` or `AIOPS_SECRET_KEY`, and an `AIOPS_SECRET_KEY_PREVIOUS` equal to `AIOPS_SECRET_KEY`.
- **Blocking**: add an approver under Pipelines → Environments → `aiops-<slug>` → Approvals and checks. An Environment created by `ado` has no check, and both the deploy and the destroy would then run unattended.
- **Pull grant**, on main: add the account id to `platform_image_pull_account_ids` in `infrastructure/iac/envs/<host-env>/env.hcl` (`<host-env>` = our sandbox host environment), then run `infra-terragrunt.yml` with `env=<host-env>`, `action=plan` and then `action=apply`, for `unit=ecr` and again for `unit=ecr-charts`. Removing the id and re-applying is the per-customer revocation lever: new versions stop, the version already cached keeps running. Azure Repos runs no PR plan, so the manual plan is the check.

### Step 5 — Vendor: deploy {#step-5-vendor-deploy}

Run `deploy-customer.yml` from branch `customer/<slug>` with `customer=<slug>`, `imageTag=<version>` (a tag `build-ecr.yml` pushed — plain `0.1.101` or `0.1.101-<suffix>.<buildId>` with the pipeline's fixed build-tag suffix; empty = the version in `pyproject.toml`) and `bootstrapDb=true`, in a window booked with the customer.

- Stage *Plan* checks the values file for placeholders and shows the terragrunt plan. On a first install that is the whole stack; there is no separate infra run.
- Approve on `aiops-<slug>`.
- Stage *Deploy* applies terragrunt (on a first install this creates the whole stack: VPC, EKS, RDS, image cache, VPC endpoints), resolves image digest + chart, verifies the signature (or warns while signing is off), snapshots RDS, writes kubeconfig, applies `deploy/customers/cluster-prereqs.yaml` (IngressClass `alb` + default StorageClass `gp3` — Auto Mode ships neither, and without them the Ingress is ignored and the PVC never binds), reads the RDS endpoint + master secret ARN from the instance, and runs the in-cluster `psql` bootstrap (`CREATE ROLE aiops LOGIN; GRANT rds_iam TO aiops;` + database + schema grant; the master password is read from the RDS-managed secret into the pod env and never printed; the pod may wait up to 5 minutes for the cluster's first node).
- Then `helm upgrade --install aiops oci://…/charts/aiops --atomic --wait --timeout 10m` with the values file, `image.tag`, the RDS endpoint and `config.AWS_REGION` from `env.hcl`. It waits up to 3 minutes for the ALB, prints `CNAME target: <host> -> <alb>`, and polls `https://<host>/healthz` through the ALB under the real host name (SNI and the ACM certificate still checked, so it passes before the CNAME exists) until it answers `{"ok":true,"version":"<version>"}`.
- It publishes `deploy-log.json` (image, digest, chart, snapshot, host, ALB name, commit). The EKS API endpoint is public (IAM-authenticated) because helm runs from a hosted agent.

### Step 6 — DNS {#step-6-dns}

The host owner creates ONE CNAME from the host name to the printed ALB name (`CNAME target: …`, also `alb_dns` in `deploy-log.json`), DNS-only. The deploy gate warns while public DNS does not reach the platform yet.

### Step 7 — Vendor: price map and smoke {#step-7-vendor-price-map-and-smoke}

- **Model price map**: in /ops → Platform defaults (`/ops/platform`) → Model price map, press **Fetch list prices**. A Bedrock model is priced from the public AWS Price List for the region it is called from — through the gateway's `aws_region_name` on that entry, or for a direct `bedrock/…` `AIOPS_MODEL` (e.g. `bedrock/apac.anthropic.claude-sonnet-5`, the regional price) the control plane's `AWS_REGION` — and the result says per model what was recorded or why not. Every other model is priced at its publisher's list price from models.dev (the publisher's own entry only): the reviewed router table maps `cx/`, `glm/`, `ds/` and `cc/` aliases; anything else is mapped from **Needs attention → Map**. A customer contract or discounted rate is a manual entry with its source note (it outranks the list price). A workspace on its own endpoint shows "Not priced yet" until its models are mapped under **Workspace endpoints** — tell the customer before the upgrade that a rate card they typed stops pricing new runs. Check **Needs attention** is empty for the smoke's model: with no trusted entry, a model id litellm already prices books its bundled table's figure, labelled an estimate on every cost screen; an id it does not know (a gateway alias) books $0, the cost guardrails (per-run cap, tenant-day budget) can never fire, and the smoke fails on `cost_usd=0`. api-mode runners pick the card up on their next job; no restart. The daily sync needs outbound HTTPS to `pricing.us-east-1.amazonaws.com` and `models.dev` (control plane only; runners never call either); without them set `AIOPS_PRICE_SYNC_INTERVAL_S=0` (or only `AIOPS_MODELS_DEV_PRICES=false` for the models.dev part) and record manual prices. To reconcile tokens later: `python -m app.aiops.price_sources reconcile --days 7` in the control-plane pod (needs `cloudwatch:GetMetricData`), or Cost Explorer by usage type in the customer's account.
- **Smoke**, from any machine that reaches the host: `BASE=https://<host> TENANT=<first-workspace-slug> ADMIN_PASSWORD_FILE=<file> scripts/customer-smoke.sh`. It logs in as `admin@local`, creates the workspace (owner `admin@local`) if missing, mints the webhook token (printed ONCE — pass `WEBHOOK_TOKEN=` on any later run, minting replaces the old one), fires one generic alert, waits for the investigation to finish, and fails unless a REAL model answered (per-model usage recorded, `cost_usd > 0`) with a non-empty RCA; it prints the incident URL. In remote mode the control plane never runs an investigation, so `done` here proves the runner plane, the runner's model access and region. There is no delete-workspace endpoint: use the customer's real first workspace slug. The minted token is a live ingest credential: store it in the customer's secret store straight away, and never paste the smoke output into a ticket, chat or email — share the incident URL only.

### Step 8 — Customer: first login and first connection {#step-8-customer-first-login-and-first-connection}

1. Log in at `https://<host>` as `admin@local` with `ADMIN_PASSWORD`, handed over out of band (never in the same message as the URL), and change it: /ops → Users → **Reset password**.
2. Connect their identity provider under **Settings · Account → Members & roles → Single sign-on** ([Single sign-on](/help/single-sign-on.md)), then point their alert sources at `https://<host>/webhook/<alertmanager|cloudwatch-sns|azure-monitor|…>/<slug>` with the token as `Authorization: Bearer <token>`.
3. **Keyless AWS** (the first read-role datasource): both Pod Identity roles (`aiops-<slug>-aiops`, `aiops-<slug>-aiops-runner`) carry `sts:AssumeRole` + `sts:TagSession` on any account's roles named `aiops-*` (a role named otherwise is out of reach); the guard is the estate role's trust. That role must trust BOTH ARNs with the workspace ExternalId (Keyless cloud access page, `GET /api/aiops/{tid}/platform-identity`) and allow `sts:TagSession` — the datasource Test runs on the control plane, investigations on the runners. With `AIOPS_KEYLESS_AWS_ANY_ACCOUNT: "true"` (the example values) the guided CloudShell setup needs no staff step: it creates `aiops-<workspace-slug>-read`/`-write` in the account the customer enters, the read role trusting both roles and the write role the control plane only (a role the customer creates by hand must be named `aiops-*` too). Three cases still need platform staff, through the workspace's `GET`/`PUT /api/aiops/admin/tenants/<id>/cloud-runtime`:
   - the account the platform runs in (`<customer-account-id>`, where a same-account trust alone lets the platform assume the role): publish only that account's exact read/write pair in `aws_targets`, with no reader. The shared runner (`aiops-<slug>-aiops-runner`) stays the reader, and the workspace keeps any-account mode for its other accounts;
   - a workspace with a private-network agent: its reads run on the agent's own AWS role, so publish that role as the reader, never the shared runner, and a prepared pair per account;
   - a workspace staff already set AWS up for: its reader (the runner role or a dedicated reader, never the control-plane role) and its pairs.

   Outside `<customer-account-id>` a prepared role must be named `aiops-*` with no path: the platform's IAM reaches no other name, and the API refuses such a pair.
4. **For the incident's Probable cause card**: CloudWatch alarms on the resources in scope routed to `/webhook/cloudwatch-sns/<slug>` (OK action on the same topic), the workspace harvest interval above 0 (900 s by default), `config_history: true` and the alarms' accounts and regions on the AWS read connection, and an AWS Config recorder that records the in-scope types, Lambda functions included — see aiops-inputs-needed.md, "What the Probable cause card needs".

Acceptance of the install = the smoke passing + one keyless connection test passing.

## 3. Upgrade {#3-upgrade}

`build-ecr.yml` on `main` publishes `<version>-<suffix>.<buildId>` (a fixed build-tag suffix set in that pipeline), `<version>`, the chart `charts/aiops:<version>` and the signature. The repositories are IMMUTABLE: the plain `<version>` tag and the chart are pushed by the FIRST build of a version only, and later builds of the same version skip them with a warning. To ship a rebuild, deploy `imageTag=<version>-<suffix>.<buildId>`; the chart stays the first build's, so a chart change needs a version bump. Then `deploy-customer.yml` from `customer/<slug>` with `customer=<slug>`, `imageTag=<new version>` (`bootstrapDb=false`). Rebase `customer/<slug>` on main first, or merge the release into it, for example `git merge v<new version>` (a merge needs no force-push): a run uses the pipeline files of the branch it is queued from, so a stale branch deploys with stale pipeline steps, and an empty `chartVersion` resolves to the version in that branch's `pyproject.toml`. Read the *Plan* stage log before approving: an application upgrade should show "No changes" in every unit, at most output-only changes. The *Deploy* stage runs a fresh `terragrunt run-all apply` without asking again, so it applies what it plans at that moment, not the plan you read. The pipeline always takes an RDS manual snapshot first and waits for it; the snapshot is named after the tag being deployed, dots as dashes: `aiops-<slug>-pre-<tag>-<buildId>`, for example `aiops-rehearsal-pre-0-1-108-17490`. Schema changes are the app's additive automigrate at boot (`app/db.py`): the old image runs against the newer schema, so a Helm rollback needs no schema rollback. Expect about a minute of downtime (single replica, Recreate).

When the release notes say the deployer policy changed, send the new zip first: the customer re-runs `bootstrap.sh apply` with the same ExternalId, and it updates only the permissions.

The deploy log artifact `deploy-log-<slug>` on every run is the record of "what is live where"; the app itself reports its version on `/healthz`.

Recovery objectives, what the snapshot does and does not cover, and the maintenance windows: [availability.md](/help/ops-availability.md).

## 4. Rollback {#4-rollback}

- **Manifests / image**: `--atomic` already rolled the release back if the upgrade failed or the pods never became ready. For a bad release that did come up: as a principal listed in `eks_admin_principal_arns` (the deploy role trusts only our pipeline), `aws eks update-kubeconfig --name aiops-<slug>`, `helm -n aiops history aiops`, `helm -n aiops rollback aiops <previous revision> --wait`. Or re-run `deploy-customer.yml` with the previous `imageTag` AND the previous `chartVersion` (after the release merge an empty `chartVersion` resolves to the new chart). This is cleaner: it snapshots again and re-verifies the signature. Helm records it as a new upgrade revision, not as a rollback, and its snapshot is named after the version being deployed, so it holds the newer version's database. Rehearsed on 7 October 2026: 0.1.105 → 0.1.108 → 0.1.105 → 0.1.108, each run about 7 minutes from approval, logins and workspaces intact. The release also carries the secret-store keys: once a key rotation has deployed, never `helm rollback` to a revision older than the last rotation deploy, and re-run a rotation deploy that `--atomic` rolled back (§12.1, rollback rule).
- **Data**: only if the new version wrote something the old one cannot read (automigrate is additive, so this is the exception). RDS restore creates a NEW instance, and the settings terraform gave the original do not follow it, so name them: `aws rds restore-db-instance-from-db-snapshot --db-instance-identifier aiops-<slug>-pg-restored --db-snapshot-identifier <snapshot from the deploy log> --db-subnet-group-name <same> --vpc-security-group-ids <same> --db-parameter-group-name <same> --multi-az --enable-iam-database-authentication --deletion-protection --backup-retention-period 7 --copy-tags-to-snapshot --manage-master-user-password` (`--no-multi-az` when `rds_multi_az = false`), and wait until it is `available`. The deploy always reads the endpoint of the instance named `aiops-<slug>-pg` and passes it as `--set-string externalPostgresql.host`, over the values file — so swap the names rather than editing the host: `aws rds modify-db-instance --db-instance-identifier aiops-<slug>-pg --new-db-instance-identifier aiops-<slug>-pg-old --apply-immediately`, wait until it is `available` under the new name, then the same call renames `aiops-<slug>-pg-restored` to `aiops-<slug>-pg`.
  - **Login**: the restored instance has a new resource id (`DbiResourceId`), and `rds-db:connect` is authorised against that id. The template grants user `aiops` on any instance of the account (`iam` unit), so the control plane logs in to the restored one unchanged. An install rendered before that grant pins the original id and is locked out after the swap (`PAM authentication failed`): bring its `iam/terragrunt.hcl` up to the template's and apply `unit=iam` (`infra-terragrunt.yml`) before the swap.
  - **Restart without terraform**: as a principal listed in `eks_admin_principal_arns`, `helm -n aiops rollback aiops <previous revision> --wait` (the bad release) or `kubectl -n aiops rollout restart deployment/aiops` (same release; otherwise pooled connections stay on the old instance). Not a deploy run: it applies every terraform unit first, and terraform still tracks the ORIGINAL instance by its resource id (the AWS provider's state id), so its plan renames `aiops-<slug>-pg-old` back to `aiops-<slug>-pg` and collides with the restored one.
  - **Before the next deploy**: reconcile the `rds` unit's state by hand — remove the original instance from it and import the restored one at the same address — then delete `aiops-<slug>-pg-old` (deletion protection off) once sure. No pipeline action does this yet, and the sequence has not been drilled.
- In their account today: the cached images (the pull-through-cache repository keeps serving them), the Helm release history, the RDS snapshots. The chart is pulled from OUR registry (`helm pull oci://…/charts/aiops --version …`, their account is granted pull). The Terraform state lives ONLY in our bucket — there is no copy in their account; the break-glass pack of brief §3.6 (state copy, module mirror, values) is not built.

## 5. Teardown {#5-teardown}

1. Agree the date with the customer, and whether they want the final database snapshot (it is kept either way; they delete it).
2. Rebase `customer/<slug>` on main, or merge main into it (a branch cut before `destroy-customer.yml` existed cannot queue it), then queue `destroy-customer.yml` from that branch with `customer=<slug>` and `confirm=<slug>`. Stage *Plan* cannot be unticked under "Stages to run"; it stops before any AWS call unless both match, the slug is a valid one and `env.hcl` names a customer deploy role (the sandbox env has none, so it can never be the target), then shows `terragrunt run-all plan -destroy`. Approve on `aiops-<slug>`: the approval exists only once someone has added it to the Environment (§2, step 4). Stage *Destroy* runs the same guard again, then, each phase with fresh credentials and a fresh AssumeRole:
   - A: `helm uninstall` (waits until the Ingress finalizer lets go, i.e. the ALB is gone), deletes namespaces `aiops` and `aiops-agents`, waits until no PersistentVolume is left (the EBS volume is deleted with it), switches RDS deletion protection off, and deletes the deploy's `aiops-<slug>-pre-*` snapshots. The Kubernetes part runs only while the cluster is `ACTIVE` and still holds the deploy role's access entry; a re-run after a failed cluster delete skips it with a warning;
   - B: `terragrunt run-all destroy`, retried once after 120 s in a task of its own (ENIs of terminated Auto Mode nodes linger); RDS writes terraform's final snapshot `aiops-<slug>-final-…` on the way out;
   - C: a read-only leftover check that fails the run while the cluster, the RDS instance, the VPC, an EBS volume of the platform or an Elastic IP tagged `Customer=<slug>` is left; only then is our state prefix `envs/<slug>/` removed. The run then prints what is kept and the follow-ups.

   A failed or timed-out run is re-run as is; the state stays until C is green.
3. The vendor, on main: remove the account id from `platform_image_pull_account_ids` and apply `ecr` and `ecr-charts`; delete the variable group and the Environment; keep or delete branch `customer/<slug>`.
4. Customer: `bash deploy/customers/bootstrap.sh delete` in CloudShell opened in the install region (with the same `AIOPS_ACCOUNT_ID`, plus `AIOPS_HOST` to delete the certificate it requested, once no load balancer uses it). It refuses while an `aiops-*` cluster, an `aiops-*-pg` instance or an `aiops-*-vpc` VPC exists in ANY enabled region (a region the organisation's SCPs close is skipped and named): a teardown that stopped half-way needs the role to finish, and a role recreated later would not get back into the cluster (EKS access entries do not follow a new role of the same name). Then they remove the DNS records and, once sure, the final snapshot.

Kept after the teardown: terraform's final DB snapshot, and the EKS secrets KMS key until its scheduled deletion date. What they cost is in §6.

## 6. Costs and timings {#6-costs-and-timings}

| Item | Time | Cost |
|---|---|---|
| Step 2, bootstrap (plan + apply) | 14 s + 19 s | not measured |
| Step 4, render + ado + pull grant | not measured (the pull grant already existed) | not measured |
| Step 5, first deploy (stack + platform) | about 27 min from approval to the HTTPS health gate (infrastructure apply 18 min 21 s, Helm install 2 min 17 s) | not measured |
| Step 7, smoke (one real-model investigation) | 34 s | not measured |
| Running stack, per day | — | not measured |
| Teardown (destroy pipeline) | not measured | not measured |
| Kept after teardown (final snapshot, KMS key) | — | not measured |

Times: the 6 October 2026 rehearsal on build 0.1.105, as in §0 ([02-install.md](/help/ops-install.md#timings-and-scope-of-proof)); no cost has been measured yet. A list-price estimate for these defaults (about USD 725–755 a month before model usage) is in [Expected monthly cost](/help/ops-expected-cost.md#expected-monthly-cost).

What the running stack is made of: the EKS control plane, Auto Mode nodes and the Auto Mode fee, one NAT gateway, RDS `db.t4g.medium` Multi-AZ with 20 GB gp3 and 7 days of backups, one ALB, VPC endpoints (an S3 gateway endpoint, free; interface endpoints for ECR api, ECR dkr, Secrets Manager, STS and the Bedrock runtime, each in two AZs at about USD 7–10 per AZ a month plus a per-GB charge), public IPv4 addresses, CloudWatch logs and alarms, and model usage per investigation.

**Availability knobs** (`env.hcl`, applied by the next deploy): `rds_multi_az` (default `true` — a synchronous standby in the second AZ, about twice the instance cost; the sandbox runs `false`), `rds_max_allocated_storage` (default `100` GiB — the autoscaling ceiling over the 20 GB start, billed only as it grows), `single_nat_gateway` (default `true` — one NAT for both AZs; `false` adds a second NAT gateway for AZ-resilient egress), `enable_core_endpoints` (default `true` — the S3, ECR, Secrets Manager and STS endpoints, about USD 60–80 a month for the four interface endpoints; `false` sends that traffic through the NAT) and `enable_bedrock_endpoint`. The RDS unit also creates two CloudWatch alarms — free storage below 2 GiB, connections above 80 % of the class limit — notifying the customer's topics in `alarm_sns_topic_arns` and, when `alarm_email` is set, the topic the unit creates, `aiops-<slug>-alarms`, with that address subscribed (SNS emails it a confirmation link that must be clicked once; at this volume the topic costs nothing). Both empty (the default): the alarms exist, nobody is paged.

**Cost guardrails** (`enable_cost_guardrails`, default `false`): a monthly AWS Budget of `cost_budget_monthly_usd` that mails `alarm_email` at 80 % and 100 % of the actual spend and at 100 % of the forecast, and a daily Cost Anomaly Detection summary of anomalies worth USD 100 or more. Only in an account that owns its billing: in a member account of an AWS Organization, budgets and anomaly detection belong to the payer, the APIs answer AccessDenied and the deploy fails — the payer sets them there. An account holds one AWS-services anomaly monitor; when it has one already, put its ARN in `cost_anomaly_monitor_arn` and the subscription uses it.

## 7. Not covered {#7-not-covered}

- **Option B**, the customer running the install themselves (CloudShell hands over to a CodeBuild job in their account, since CloudShell ends idle sessions): on request; it moves the Terraform state, secrets and upgrades to their side.
- **GitHub Actions** in the customer's GitHub: on request (needs a GitHub OIDC trust on our infra role and the install extracted from the pipeline).
- **Image mirror / tar export** (each release pushed into their registry): on request; the pull-through cache is the default.
- **ACR, Azure / GCP accounts**: the trust chain (workload identity / Entra app), the registry path (ACR's cache cannot pull from a private ECR) and the signing-key location differ.
- **The permissions boundary** on `aiops-deployer` (§1.4).
- **Private EKS endpoint** (needs a self-hosted agent in their VPC), an internal ALB, ECR cross-account **replication** (the no-NAT fallback), in-cluster admission verification (their Kyverno/Ratify; we hand over `aiops-release.pub` on request once signing is on).
- **Air gap** = profile 2 (Zarf package, `--verify always`), design only; its real prerequisite is an LLM endpoint inside the boundary.
- **Licence enforcement**: no key issued yet, so every install runs unlicensed — /ops → Platform shows "Unlicensed — no enforcement" and nothing is paused. The slot is wired (§11).
- **Image signing**: off until the signing key exists; both pipelines then warn that images are unsigned/unverified (§9).
- **Digest pinning in Helm**: the pipeline verifies and records the digest but the chart has no `image.digest` value, so the pod pulls by tag (one template line closes it). **Reproducible image builds**: kubectl and helm are pinned, but the `python:3.12-slim` base tag and the Debian security updates applied at build time float, so two builds of one commit can differ — pin the base digest before a customer asks us to reproduce a signed digest.

## 8. Proof {#8-proof}

- **Dry run, 2026-09-28**, in our trial account in `ap-southeast-1`, which is not a fresh account (service-linked roles, quotas and Bedrock state already existed there, beside other workloads), with the earlier by-hand steps: install, `/healthz` gate, HTTPS through the ALB, an upgrade through the pipeline and a `helm rollback` all passed. It took four pipeline runs, each fixing a real defect now in this lane (the trust statement split, the ECR pull-through service-linked role, the db bootstrap without `kubectl attach`, no `$` in the bootstrap SQL). The smoke passed on Nova Pro; AWS refused Anthropic models for that account. The teardown was done by hand; `destroy-customer.yml` is meant to replace it, and its first run is the rehearsal below. What only a brand-new account shows (first-use service-linked roles, a small vCPU quota, Bedrock with nothing on file) is proven only by the customer's own first install.
- **Rehearsal of this runbook**, 6 October 2026, build 0.1.105: a fresh install into a separate AWS account under the slug `rehearsal` — bootstrap plan and apply, deploy and the real-model smoke, timed in §0 and §6 ([02-install.md](/help/ops-install.md#timings-and-scope-of-proof)). The certificate was already issued and the pull grant already existed, so neither was timed. The same install took the §3 upgrade and the §4 rollback on 7 October (0.1.105 → 0.1.108 → 0.1.105 → 0.1.108) and a point-in-time restore drill on 8 October (record). No record yet of bootstrap `delete` or of a teardown through `destroy-customer.yml` with its leftover check, so §0 row T and every cost in §6 stay unmeasured.

## 9. One-time setup on our side (not per customer) {#9-one-time-setup-on-our-side-not-per-customer}

- The CI role policies are applied by hand, not by terraform: `infrastructure/iac/oidc-role/infra-role-policy.json` on `aiops-demo-ado-infra` (its `AssumeCustomerDeployRole` statement allows `sts:AssumeRole` and `sts:SetSourceIdentity` on `role/aiops-deployer` only) and `role-policy.json` on the build role.
- Signing (optional until the key exists): create the KMS key once — `aws kms create-key --key-spec ECC_NIST_P256 --key-usage SIGN_VERIFY` + `aws kms create-alias --alias-name alias/aiops-release --target-key-id <id>`; re-apply `role-policy.json` (`CosignReleaseKey`) to the build role; export `cosign public-key --key awskms:///alias/aiops-release > deploy/customers/aiops-release.pub`; set `COSIGN_KMS_ALIAS: alias/aiops-release` in BOTH `build-ecr.yml` and `deploy-customer.yml`. Where that key lives follows the registry-host decision (decision log §6.4).

## 10. Sandbox notes that affect this path {#10-sandbox-notes-that-affect-this-path}

- **Demo sync window.** On the demo platform (ArgoCD, not this pipeline) the AppProject `aiops` (`infrastructure/gitops/apps/aiops/project.yaml`) denies automated syncs of the `aiops` and `aiops-cloud-proof-primary-reader` Applications **09:00–10:30 UTC Mon–Fri** — the customer's daily sync, where a push-triggered Recreate rollout cost 11 minutes on 01-Oct. image-updater still records the new tag during the window; the rollout runs when it closes. A hotfix inside the window is a manual **Sync** in ArgoCD (allowed by the window). Commits that touch only `infrastructure/` do not build an image and roll nothing. The customer path is untouched: `deploy-customer.yml` is run by hand, so its timing is yours.
- The sandbox `db-init` Job (`infrastructure/gitops/apps/db-init/`, hardcoded RDS host, rode ArgoCD) was **removed on 06-Oct-2026**: the databases and roles exist, and the Job had been failing on every sync since the RDS-managed master password (`manage_master_user_password`) rotated away from the copy in the `aiops-db` Secret. A cold rebuild of the sandbox needs it back: restore the folder from git (last present at `6b294eb3`) after copying the current RDS-managed master password (`rds!db-…` secret) into `RDS_MASTER_PASSWORD` in Secrets Manager `aiops-demo/database`. The customer path never used this Job — `deploy-customer.yml` (`bootstrapDb`) creates the role itself.
- `infrastructure/gitops/values/aiops.yaml` still pins build `0.1.0-<suffix>.15787` as the bootstrap tag (image-updater writes back to ArgoCD, not git); irrelevant to customers, but a cold ArgoCD bootstrap of the sandbox starts ~80 releases behind.

## 11. Licence {#11-licence}

An offline signed file, no phone-home ([licence.md](/help/ops-licence.md)): the pod verifies the JWT in `AIOPS_LICENSE` against `licence.publicKey` at boot and every scheduler tick. **No key exists yet** (`alias/aiops-licence` waits on the vendor-of-record decision, decision log §6): every install runs `unlicensed` = no enforcement. The slot is wired, so that changes with one variable.

- **Install**: `AIOPS_LICENSE` is a secret variable of `aiops-<slug>-deploy` (`mint` writes it empty; an older group gets it from `ado <slug> --update-group`); the deploy passes it as `--set-string licence.value`. The public key is `licence.publicKey` in the values file on `customer/<slug>`.
- **Renew**: the new token as `AIOPS_LICENSE` in `~/.aiops/customers/<slug>.env`, `ado <slug> --update-group`, then `deploy-customer.yml` with the current `imageTag` — a rollout, no endpoint. The file is the source of truth (Step 4): a token pasted only in the ADO Library goes back to the file's value at the next `--update-group`.
- **Issue**: `scripts/issue-licence.py` ([licence.md](/help/ops-licence.md)). `kms:Sign` on the alias belongs to the role of an approval-gated issue job only, never to the build role, which every push to main and the power schedule assume. That job and its role are not built yet.

| When | State | Ops console | Runs |
|---|---|---|---|
| before `exp` | `valid` | card only | normal |
| `exp` .. `exp + grace_days` | `grace` | banner: expired, N days of grace left | normal — a warning, nothing paused (`grace_days: 0` in the file = no grace at all) |
| after `exp + grace_days` | `expired` | red banner | new agent runs and approved executions refuse to start |
| any signature / claim failure | `invalid` | red banner with the reason | same as `expired` |
| no licence, or no public key | `unlicensed` | card says "no enforcement" | normal (warning in the log at boot) |

## 12. Rotating secrets {#12-rotating-secrets}

The secrets file `~/.aiops/customers/<slug>.env` stays the source of truth (Step 4): change a value there, run `ado <slug> --update-group`, then `deploy-customer.yml` from `customer/<slug>` (rebased on main) with the live `imageTag`. A changed Secret restarts the control plane through its checksum, and the provisioner recreates every runner whose environment changed.

| Secret | Where it lives | How to rotate | Effect | Who |
|---|---|---|---|---|
| `SECRET_KEY` | secrets file → `secrets.AIOPS_SECRET_KEY` | a new value (`python3 -c 'import secrets; print(secrets.token_urlsafe(48))'`), update-group, deploy | it signs only the 10-minute SSO sign-in state and the 24-hour SSO logout hint; sessions are stored hashed in the database, so nobody is signed out — an SSO sign-in in progress starts again | the vendor |
| `ADMIN_PASSWORD` | the bootstrap superadmin, seeded only when that account is created | change it in Ops › Users, then mirror it in the secrets file | the file value is ignored once the account exists; mirroring keeps the file true for a rebuild | the vendor or the customer's superadmin |
| `AIOPS_SECRET_KEY` | secrets file → `secrets.AIOPS_TENANT_SECRET_KEY`; encrypts every stored credential | §12.1 — never by overwriting the value | two deploys; connections keep working throughout | the vendor; the customer approves the deploys |
| `AIOPS_API_KEY` | secrets file → `secrets.AIOPS_API_KEY` (empty on Bedrock) | a new key at the gateway, then the file, update-group, deploy; revoke the old key at the gateway afterwards | control plane restart; runners recreated with the new key | the vendor (gateway owner) |
| `AIOPS_LICENSE` | secrets file → `licence.value` | §11, *Renew* | a rollout, nothing else | the vendor |
| Webhook ingest tokens | per workspace; only a hash is stored | the workspace owner mints a new one in the portal and puts it in every alert source | no overlap window: the old token stops at once, so change the senders in the same sitting | customer workspace owner |
| Personal access tokens | per user; only a hash is stored | mint a new one on the Tokens page (at most 365 days), switch the client, revoke the old one | none beyond the client switch | each user |
| Runner tokens (`rt_`) | per runner; only a hash is stored | runner pods the platform provisions get a fresh token on every (re)create — nothing to do. A private-network agent: mint a new agent token in the portal (`datasources.manage`), set it on the agent, revoke the old one | the agent reconnects with the new token | customer operator |
| Datasource credentials | the tenant secret store, write-only | edit the connection and enter the new value (a blank field keeps the stored one), then Test | a new connection revision: the last Test no longer vouches for it | customer operator |
| Database | RDS IAM authentication: no database password exists | nothing to rotate | — | — |

### 12.1 The tenant secret-store key {#12-1-the-tenant-secret-store-key}

`AIOPS_SECRET_KEY` encrypts every credential the platform stores. A rotation keeps the old key next to the new one as `AIOPS_SECRET_KEY_PREVIOUS` (it still decrypts, never encrypts) until the control plane has re-keyed every stored secret, which it does at boot: one compare-and-swap per row, a `secret.rekeyed` audit row per secret in its workspace's audit log, one `secrets.rekey_completed` summary in the platform audit log. A secret no configured key opens is never touched, and an edit of that connection answers 409 instead of overwriting it. The re-key needs a build with this support, deployed once before the first rotation: that boot stamps every stored secret with the current key.

1. **Precheck.** `AIOPS_SECRET_KEY_PREVIOUS` is empty in the secrets file (no rotation open). Copy the secrets file, at least the current key, to the password vault. Take the census on the install — an EKS admin principal (the customer's, §4) runs it and sends back the JSON, which holds counts, a key fingerprint and opaque secret ids, never a secret:

   ```bash
   kubectl -n aiops exec deploy/aiops -- python -m app.aiops.secrets rekey --dry-run > census.json
   ```

   It must show `"remaining": 0` and `"unreadable": []`; note `readable`. Each `deploy-customer.yml` run takes an RDS snapshot before the helm upgrade.
2. **Rotate.** `python3 scripts/customer-onboard.py rotate-key <slug> --census census.json` — it refuses without a clean census of the current key, then makes the current key `AIOPS_SECRET_KEY_PREVIOUS` and mints a new `AIOPS_SECRET_KEY`. Copy the file to the vault again: it holds the only copy of the new key. Then `ado <slug> --update-group` (a human with `ADO_PAT`).
3. **Deploy with both keys.** Rebase or merge `customer/<slug>` with main (the mapping of the previous key lives in the branch's `deploy-customer.yml`), then queue `deploy-customer.yml` with `customer=<slug>` and the live `imageTag`; the customer approves. The control plane restarts with both keys and re-keys at boot; db-mode runners restart with both keys.
4. **Check.** Ops › Audit log, the newest `secrets.rekey_completed`: `key_id` is the new fingerprint `rotate-key` printed, `remaining` 0, `unreadable` [], `readable` the census count (give or take connections added or deleted since). Each workspace's audit log shows `secret.rekeyed` per stored secret. Test the connections.
5. **Retire.** Save that event as JSON (copy it from Ops › Audit log, or `GET /api/aiops/admin/audit?action=secrets.rekey_completed&limit=1` with a superadmin token), then `python3 scripts/customer-onboard.py retire-key <slug> --rekey-report event.json`. It refuses unless the event is for the new key with `remaining` 0 and `unreadable` [], then moves the previous key to `AIOPS_SECRET_KEY_RETIRED_<YYYYMMDD>`, a line that never goes to Azure DevOps. `ado <slug> --update-group`, deploy again, Test again.

**Stop rule.** `unreadable` not empty or `remaining` above 0: do not retire. The usual cause is a previous key that never reached the pod (a customer branch not rebased, or update-group skipped): the secrets stay untouched and the connections fail closed. Fix the plumbing and deploy again — the boot re-key resumes — or roll back: swap the two values in the secrets file (the old key back as `AIOPS_SECRET_KEY`, the new one as `AIOPS_SECRET_KEY_PREVIOUS`), update-group and deploy; the boot re-key moves whatever was written under the new key back to the old one, and step 5 closes the rollback like any rotation.

**Rollback rule.** The release carries the keys (they ride `--set-string` into the chart Secret), so rolling it back can undo a rotation step. A `helm rollback` to a revision from before step 3, or `--atomic` undoing a step-3 run that failed after its pod had booted and re-keyed, brings the old key back alone. Every secret already re-keyed then fails closed (connections down, edits 409) until the next pipeline deploy, and a credential saved meanwhile is written under the old key: once that key is retired, only a deploy carrying the retired line as `AIOPS_SECRET_KEY_PREVIOUS` opens it again (as in the DR rule), and step 5 then closes that like any rotation. From step 3 on:

- roll back only by re-running `deploy-customer.yml`, which renders the current keys from the group (to undo the rotation itself, swap the keys as the stop rule says); never `helm rollback` to a revision older than the last rotation deploy (step 3 or step 5);
- a step-3 run that failed is re-run, and must succeed, before step 4;
- retire only on the newest `secrets.rekey_completed`, written after the last successful deploy run started: an older event does not count what was saved under the old key since.

**DR rule.** Keep the `AIOPS_SECRET_KEY_RETIRED_<YYYYMMDD>` line until the oldest snapshot and point-in-time-restore window from before the rotation have expired: a restore of such a backup needs that key — deploy the restored install with it as `AIOPS_SECRET_KEY_PREVIOUS` and the boot re-key moves its secrets onto the current key.

db-mode runners restart at each deploy of a rotation (their environment changes); api-mode runners never receive either key.

**Compose.** The same steps with the `.env` file as the source of truth:

```bash
docker compose exec aiops python -m app.aiops.secrets rekey --dry-run  # census: remaining 0, unreadable []
# .env: AIOPS_TENANT_SECRET_KEY_PREVIOUS=<the old key>, AIOPS_TENANT_SECRET_KEY=<a new key>
docker compose up -d                                                    # restarts with both keys, re-keys at boot
docker compose exec aiops python -m app.aiops.secrets rekey --dry-run  # remaining 0, unreadable [] on the new key
# .env: clear AIOPS_TENANT_SECRET_KEY_PREVIOUS (keep the old key in the vault), then
docker compose up -d
```

Runner containers the control plane provisioned itself (`AIOPS_RUNNER_PROVISIONER=docker`) keep the environment they were created with. After each `up -d`, stop them with their drain window and remove them; the control plane provisions them again with the current keys:

```bash
ids=$(docker ps -aq --filter label=aiops.managed=1); [ -z "$ids" ] || { docker stop -t 120 $ids; docker rm -f $ids; }
```


## Next steps

- Next: [Licence](/help/ops-licence.md)
- [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt)
- [Documentation home](/help/)