# Availability and recovery

> As of 0.1.110 (8 October 2026): what a customer can rely on today, and what is roadmap.

_Reference · Updated 2026-10-24 · Describes release 0.1.164_

As of 0.1.110 (8 October 2026): what a customer can rely on today, and what is roadmap.

The control plane runs as **one pod** behind the ALB health check: when the pod crashes or is evicted, Kubernetes replaces it in roughly 30–60 s; when its node is lost, the replacement first waits for Kubernetes to give up on the node and release the `/data` volume — up to about 12 minutes, not yet measured. Every upgrade is a `Recreate` rollout with about a minute of downtime. The agents run in a separate runner pod per workspace and every job is durable: a runner that dies mid-investigation is re-queued when its lease expires (≤ 3 minutes) and the next runner resumes from the per-step committed transcript, up to 3 attempts. An api-mode runner (the customer and demo default) checkpoints every step through the control plane, so a step that checkpoints while the control plane is down — during an upgrade too — is lost the same way and the run waits out its lease. The database is RDS PostgreSQL, Multi-AZ in the customer template, with 7 days of automated backups (point-in-time restore) plus a manual snapshot before every upgrade. The control plane's `/data` volume lives in one Availability Zone, so losing that zone stops the control plane until the zone returns, whatever the database does. Leader election for a multi-pod control plane exists (`app/aiops/leader.py`, a Postgres advisory lock) and is on in the demo, but the platform ships one replica because the per-tenant lake sits on a single-node volume. A multi-replica, multi-AZ control plane is roadmap, not a setting.

## Recovery objectives {#recovery-objectives}

| Failure | RPO | RTO | Notes |
|---|---|---|---|
| Control-plane pod crash or eviction | 0 for committed state; one step for an api-mode run that checkpoints in the gap | ~30–60 s; that run resumes as its next attempt once its 180 s lease expires | demo ALB: check every 5 s, 2 healthy, 10 s drain; an api-mode runner retries a refused checkpoint for about 3.5 s, then drops the attempt |
| Upgrade (`Recreate`) | as a pod crash | ~1 min, planned; the same lease wait for such a run | db-mode runners keep going: the DB rows are the queue. From 0.1.149 the start does not grow with the data: before serving, the pod only adds tables and columns (a default in the catalog, no row rewritten, each statement giving up a table lock after 3 s and retrying); indexes and data fixes run after it serves (`CREATE INDEX CONCURRENTLY`, one pod at a time) |
| Node loss | 0 for committed state | up to ≈ 12 min, not yet measured | pods leave an unreachable node only after Kubernetes' default 300 s not-ready/unreachable toleration (the chart sets no shorter one), then the ReadWriteOnce `/data` volume waits up to 6 min for a forced detach; a runner on that node is evicted on the same 300 s clock, so its jobs resume after about 6 min, not 3 |
| Runner pod mid-job | one step (per-step transcript commit) | ≤ 3 min lease, then resume | `aiops_job_lease_s=180`, `aiops_job_max_attempts=3` — a run out of attempts fails and its incident is escalated; chat, benchmark and API-facade runs are not resumed |
| RDS instance or host | 0 (synchronous Multi-AZ standby) | ~1–2 min, automatic failover | single-AZ (`rds_multi_az = false`): no standby, so recovery takes longer, and lost storage means a restore, as below |
| Bad data or a bad release (restore) | up to about 10 min (PITR: RDS ships transaction logs about every 5 minutes), or the pre-upgrade snapshot; measured 8 min 14 s at the restore point on a customer-template install (samples from 3 min 03 s to 8 min 14 s) and 2 min 34 s to 4 min 12 s on the hosted sandbox | tens of minutes — a restore creates a NEW instance, which then takes over the original's name. Measured on the hosted sandbox (7 October 2026, 20 GB): 11 min to a restored copy, 14 min to a verified one (drill record); on a customer-template install (8 October 2026, 20 GB): 19 min to a restored copy, 22 min to a verified one (drill record); the cut-over is not drilled. The customer template grants the database login (`rds-db:connect`) to user `aiops` on any instance of its account (`dbuser:*/aiops`, `iam` unit), so the control plane logs in to the restored instance unchanged; the hosted sandbox, and an install rendered from an older template, pin the original instance's resource ID, which a restored instance does not share, so there the `iam` unit must be applied again first | runbook §4 Rollback |
| Availability Zone | 0 for the database (Multi-AZ) | database ~1–2 min; the control plane is down until the zone returns when its `/data` volume is in that zone | egress runs through one NAT gateway unless `single_nat_gateway = false` |
| DuckDB lake on `/data` | not backed up | re-collect from the sources | see below |

### Measured on the demo cluster (3 October 2026, 0.1.104) {#measured-on-the-demo-cluster-3-october-2026-0-1-104}

Two runner-kill drills on a throwaway workspace with a real model:

| Scenario | What happened | Time to completion |
|---|---|---|
| Planned: `kubectl delete pod` on the runner (rollout, drain) | the old process finished the run inside its 300 s drain while the replacement pod was already up; the attempt counter stayed 0 | 119 s after the delete |
| Unplanned: hard kill (`--grace-period=0 --force`) at step 2 | the replacement pod was up in 3 s, waited out the 180 s lease, then resumed from the committed transcript as attempt 1 (steps 2 → 8) | 301 s after the kill |

No run was lost or duplicated and the incident kept one history. Node loss has not been drilled; its row above is an estimate.

## Database: the RDS settings {#database-the-rds-settings}

From `infrastructure/iac/envs/_customer-template` (`rds/terragrunt.hcl`, `env.hcl`).

| Setting | Value | Knob |
|---|---|---|
| Engine, class, storage type | PostgreSQL 16, `db.t4g.medium` (~340 connections), gp3 | `rds_instance_class` |
| Storage | 20 GB, autoscaling up to 100 GB | `rds_allocated_storage`, `rds_max_allocated_storage` |
| Multi-AZ | on: a synchronous standby in the second AZ, about twice the instance cost | `rds_multi_az` |
| Automated backups | 7 days, point-in-time restore | `rds_backup_retention_period` |
| Pre-upgrade snapshot | `aiops-<slug>-pre-<tag>-<buildId>`, taken by `deploy-customer.yml` before every rollout | — |
| Snapshot tags | the stack's tags are copied to every automated and manual snapshot | — |
| Deletion | `deletion_protection`, final snapshot on destroy | — |
| Alarms | free storage below 2 GiB, connections above 80 % of the class limit; they notify the customer's SNS topics and, with `alarm_email` set, the topic `aiops-<slug>-alarms` that mails that address (a named recipient) | `alarm_sns_topic_arns`, `alarm_email` |
| Egress | one NAT gateway in one AZ | `single_nat_gateway` (`false` = a second NAT gateway, AZ-resilient egress) |
| AWS service endpoints | an S3 gateway endpoint and ECR, Secrets Manager, STS and Bedrock interface endpoints with private DNS: cached image pulls, secret reads, role assumption and model calls stay on AWS's network; the NAT still carries the first pull through the cache, webhooks and tickets | `enable_core_endpoints`, `enable_bedrock_endpoint` |

The hosted sandbox database (`aiops-demo-pg`) differs from the template: single-AZ in ap-southeast-1a while the control plane's `/data` volume is in ap-southeast-1b (losing either zone stops it), `db.t4g.medium`, PostgreSQL 16 with automatic minor upgrades. It had no automated backups until 7 October 2026; since then it keeps 7 days with point-in-time restore, deletion protection and a final snapshot on destroy. Its two alarms exist but notify no one (`alarm_email` and `alarm_sns_topic_arns` are unset in the sandbox host environment's `env.hcl`). Its windows are the ones AWS assigned (the IaC sets none): backups daily 15:41–16:11 UTC, maintenance Tuesdays 14:08–14:38 UTC. An optional operating-system update (`system-update`) is pending and is not applied before the handover. Upgrades there roll through ArgoCD image-updater and take no pre-upgrade snapshot.

Authentication is IAM (signed 15-minute tokens); the master password lives in Secrets Manager and is read only by the first-install step, so no database password needs backing up.

## What is NOT backed up {#what-is-not-backed-up}

The control plane's `/data` volume (ReadWriteOnce EBS) has no volume snapshot, no Data Lifecycle Manager policy and no AWS Backup plan, by decision (decision-log.md §3.6): its backup path is the database. Skills are re-materialised from their rows at every boot, and the files of command-line connections (kubeconfigs, SSH keys) are written again from the secret store the next time the connection is used, so a lost volume is replaced by a new empty one and a control-plane start. The one thing it takes with it is the per-tenant DuckDB lake, which is re-collected from its sources by the collection specs; the pilot tenant has no lake specs, so it has no lake to lose.

The volume is still credential-bearing: those connection files are plaintext copies (mode 0600, under `/data/clifiles`), named by content and never pruned, so a rotated kubeconfig or key stays next to its successor; an install that predates the database secret store also keeps its old `secrets.json.migrated` there. Treat the volume, and any copy of it, like the credentials themselves.

## Maintenance windows (demo cluster) {#maintenance-windows-demo-cluster}

ArgoCD denies automated syncs of the platform's two Applications **Mon–Fri 09:00–10:30 UTC** (`infrastructure/gitops/apps/aiops/project.yaml`; a manual Sync stays allowed for hotfixes; the cluster's other Applications are not held). Karpenter starts no voluntary node replacement (drift, consolidation) in the same window and replaces one node at a time otherwise; from 6 October 2026 until the hold is removed after the handover it replaces no node for drift or under-use at all, only empty ones (`infrastructure/gitops/apps/nodeclass/nodeclass.yaml`). Two replacements ignore every budget: an unhealthy node, and expiry — every node expires 14 days after it was created (`expireAfter: 336h`). A node being replaced keeps pods that a PodDisruptionBudget protects for up to 24 hours. Rotating the control plane's own node moves the control plane: a `Recreate`, about a minute when a node with room exists in the zone of its `/data` volume, longer when one has to start. Customer installs have no ArgoCD and their upgrade is a pipeline run scheduled by you; their nodes follow the EKS Auto Mode built-in node pool's own replacement schedule.

## Multi-replica control plane: what blocks it {#multi-replica-control-plane-what-blocks-it}

Leader election already keeps the scheduler, audit-chain sealer, job sweep, runner provisioner, the private-chat reconcile sweep and DB gauges on one elected replica (chat turns themselves run on whichever replica serves the request), and the chart refuses `replicaCount>1` without it. The platform is still single-replica because the `/data` volume is ReadWriteOnce (pins every replica to one node); the runner-token rate limit, ingest token bucket and admission gate, SSE pub/sub, chat turn locks and the licence/price caches are per-process memory; three check-then-write paths reachable from any replica are guarded only by in-process locks — IdP email-domain claims (one email domain, one workspace's SSO: two replicas could send a domain's sign-ins to the wrong workspace), compliance incident landing (twin incidents) and change-harvest recording (twin change rows); and every replica adds a connection pool (5 + 10 overflow + the elector's session) that must fit `max_connections` (three replicas exhausted the sandbox on 3 September 2026). A pooler in transaction mode is no way out: the elector's lock and the online index builder's lock are session locks, and `CREATE INDEX CONCURRENTLY` refuses to run inside a transaction, so both need a direct connection. Order of work after the 16 October handover: counters to Postgres → IdP domain claims on a normalised table with a unique index, and Postgres advisory locks or partial unique indexes for compliance landing and harvest recording → lake off the PVC and `/data` to emptyDir → SSE over LISTEN/NOTIFY → pool sizing or a pooler → `replicaCount: 2` with anti-affinity and a PDB on the demo first. Decision record: decision-log.md §3.5.

Related: [customer-deploy-runbook.md](/help/ops-installation-runbook.md) §3 Upgrade and §4 Rollback; [architecture.md](/help/architecture.md) §11 and §14.


## Next steps

- Next: [Audit trail retention policy](/help/ops-audit-retention.md)
- [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt)
- [Documentation home](/help/)