# Rollback

> Choose an image rollback for a bad release that can still read the database. Restore data only when necessary. Additive schema migration normally lets the previous image use the newer schema.

_How-to · Updated 2026-10-24 · Describes release 0.1.164_

Evidence reviewed **8 October 2026**; operating baseline **0.1.130** (the 16 October 2026 handover build; see the [release notes](/help/release-notes.md)).

Choose an image rollback for a bad release that can still read the database. Restore data only when necessary. Additive schema migration normally lets the previous image use the newer schema.

## First check the key-rotation boundary {#first-check-the-key-rotation-boundary}

The Helm release carries the tenant secret-store keys. **Never use Helm rollback to a revision older than the most recent rotation deploy.** A pod may already have re-encrypted database secrets before a failed upgrade's automatic rollback. Bringing back the old key alone then makes those secrets unreadable: connection reads fail closed and edits return 409.

Across this boundary, re-run `deploy-customer.yml` with the intended image and chart and the **current key pair** from the secret group. A failed rotation deploy must be re-run successfully before relying on its re-key summary. To undo the rotation itself, swap current and previous keys in the source secrets file and use the two-key procedure in [Security operations](/help/ops-security-operations.md#tenant-secret-store-key). Retain old keys until pre-rotation backup windows expire.

## Roll back the image and manifests {#roll-back-the-image-and-manifests}

If readiness fails during deployment, `--atomic` attempts rollback already; read the result rather than assume it succeeded. For a bad release that became ready, the normal recovery is another `deploy-customer.yml` run with the **previous `imageTag` and previous `chartVersion`**. An empty chart version uses the version on the customer branch and may deploy the new chart again. This path takes another snapshot and uses the pipeline's verification stage.

From 0.1.149 on, roll back the image and keep the newer chart: pass the previous `imageTag` with the current `chartVersion`. The newer chart's startup probe gives the pod five minutes to start; an older image still runs a backfill over every table at each start, and the older chart's liveness probe allows 140 seconds from container start. Nothing is undone in the database: the columns, indexes and data fixes the newer release made stay, and the older image does not read them. An index left invalid because the pod was stopped mid-build is harmless (queries skip it) and the next 0.1.149-or-later start rebuilds it ([Upgrade](/help/ops-upgrade-and-verify.md#schema-changes-after-the-pod-is-ready)).

For an emergency Helm rollback within the safe key boundary, an IAM principal listed in the installation's `eks_admin_principal_arns` uses runbook §4's commands. Replace the example cluster and revision with values from that installation; inspect history before choosing the revision.

```bash
aws eks update-kubeconfig --name aiops-example-org
helm -n aiops history aiops
helm -n aiops rollback aiops 3 --wait
```

The vendor deploy role trusts its pipeline only; use the customer emergency access principal for these commands. Check the active kubectl context before the Helm actions. After either rollback path repeat [Release verification](/help/ops-upgrade-and-verify.md#verify-the-release).

The rehearsal proved **0.1.105 → 0.1.108 → 0.1.105 → 0.1.108**, about **7 minutes per pipeline run from approval**, with login and workspace data intact. This is image rollback evidence, not data-restore or key-rotation rollback proof ([deployment runbook §4](/help/ops-installation-runbook.md#4-rollback)).

## Restore data when image rollback is insufficient {#restore-data-when-image-rollback-is-insufficient}

Use [Backup, restore and DR](/help/ops-backup-restore-and-dr.md) and runbook §4. The deploy artifact identifies the pre-upgrade snapshot. RDS restoration creates a **new instance**; preserve the original while checking the copy. Retain the source network, parameter group, IAM authentication, backup, encryption and deletion-protection settings explicitly.

After verifying the restored copy:

1. Confirm the database login grant for its new resource id. Current customer template grants the `aiops` user across instance ids in the account; older rendered installs need the IAM template update before a swap.
2. Follow runbook §4's controlled rename of original to `-old`, restored to the original name. The deployment reads the endpoint by instance name.
3. Restart the control plane without a deployment apply: otherwise Terraform still follows the original instance and can collide with the replacement's name. Use the correct current keys; a restored pre-rotation database needs its retained key as previous.
4. Before the next deployment, reconcile RDS state by removing the original instance and importing the restored one at the same address. This is a manual cloud/release-team task, not a delivered pipeline action.
5. Verify health, login, runner execution, connection decryption and full audit-chain replay before releasing the retained original for deletion.

The published drills verified side copies only. Snapshot restoration, name swap, Terraform state reconciliation and service cutover have **not been drilled**. Do not report the verified-copy timings as production recovery time. Record a separate cutover result when that drill is completed.

Sources: [Deployment runbook §4 and §12.1](/help/ops-installation-runbook.md#4-rollback), 7 October drill, and 8 October customer-template drill.


## Next steps

- Next: [Backup, restore and disaster recovery](/help/ops-backup-restore-and-dr.md)
- [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt)
- [Documentation home](/help/)