# Backup, restore and disaster recovery

> Plan for 5–10 minutes of potential data loss with point-in-time restore (PITR). The customer-template drill reached a verified database copy in 21 minutes 56 seconds; this excludes service cutover.

_How-to · Updated 2026-10-19 · Describes release 0.1.164_

Evidence reviewed **8 October 2026**; operating baseline **0.1.130** (the 16 October 2026 handover build; see the [release notes](/help/release-notes.md)).

Plan for **5–10 minutes of potential data loss with point-in-time restore (PITR)**. The customer-template drill reached a verified database copy in **21 minutes 56 seconds**; this excludes service cutover. The control plane stayed on its source throughout both published drills.

## What is protected {#what-is-protected}

| Item | Protection and recovery |
|---|---|
| PostgreSQL database | Customer template: Multi-AZ synchronous standby, 7 days automated backups/PITR, deletion protection and final snapshot on teardown |
| Pre-upgrade state | Deployment pipeline creates and waits for a manual snapshot before rollout; its id is in `deploy-log.json` |
| Tenant secret-store encryption keys | Vault copy of the onboarding secrets file, including current and retained previous/retired keys; a database backup does not supply them |
| Skills and CLI connection files on `/data` | Rebuilt from database rows: skills at boot; connection files on next use |
| DuckDB lake on `/data` | Not backed up; re-collect from source systems using collection specifications |
| Deployment definitions and recovery state | Customer branch and vendor Terraform state; no delivered customer-side state-copy/break-glass package |
| Record archive bucket (optional, `AIOPS_ARCHIVE_BUCKET`) | Not a database backup: it holds closed records after retention removed them from the database, one object per workspace-month, Object Lock governance 5 years, versioned. Each manifest lists the records it holds, so the `archivedrecord` index can be rebuilt after a database restore (no tool ships for it yet). The objects are readable only with the bucket's KMS key: `prevent_destroy` and a 30-day deletion window protect it, and losing the key makes the archive unreadable. A database restored to before a batch's removal brings the rows back alongside their archived copy; the next sweep archives them again under a new batch id |

Daily posture snapshots introduced in 0.1.110 are PostgreSQL rows and are covered by the database backups; they do not depend on `/data`. This storage coverage is not a separate restore-drill result.

The decision for `/data` avoids extra plaintext credential copies in volume snapshots. Its rebuild is code-backed, not yet drilled. Revisit the lake decision when a workspace depends on retained lake history that sources cannot re-create.

## Recovery objectives and measured evidence {#recovery-objectives-and-measured-evidence}

| Failure or check | RPO | RTO / evidence |
|---|---|---|
| Control-plane pod crash | Committed state retained; an API-mode run may lose one checkpoint step | Approximately 30–60 s pod replacement; affected run waits its 180 s lease |
| Runner crash | At most the uncommitted step | Lease at most 180 s, then eligible work resumes, at most three attempts |
| RDS Multi-AZ failover | 0 for synchronously committed database state | Approximately 1–2 min; availability estimate, not the copy-restore drill |
| PITR | Plan 5–10 min; sampled lag varies | New instance plus verification, then a separate cutover |
| Pre-upgrade snapshot restore | Snapshot boundary | Documented procedure; snapshot restore and cutover not drilled |
| Node or zone loss | Committed database state retained with Multi-AZ | Node-loss estimate up to about 12 min; `/data` zone loss can hold the control plane down until the zone returns |
| 7 October hosted-sandbox copy drill, 20 GB | Lag samples 4m12s and 2m34s | Available 11m08s; verified 14m11s |
| 8 October customer-template copy drill, 20 GB, 0.1.108 | Lag samples 8m14s, 5m51s and 3m03s | Available 19m24s; verified 21m56s, including 9m15s of initial copy backup |

The 7 October record and 8 October record checked identical row counts at the restore point and complete audit-chain replay. The latter checked both chains on the customer-template copy. Each copy was deleted afterwards. These receipts prove copy restoration and integrity, **not cutover, RUN-02 end-to-end recovery, a `/data` rebuild or decryption of every stored connection credential**.

## Restore a side copy for verification {#restore-a-side-copy-for-verification}

The following AWS operations quote the tested **How to repeat** procedure in the 7 October record, also used on 8 October. Example names replace the record's environment names. Run as an approved cloud operator in the source account and region. Set `AIOPS_DB_SUBNET_GROUP` and `AIOPS_DB_SECURITY_GROUP` to the source's private groups and `AIOPS_RESTORE_TIME` to its recorded latest restorable UTC timestamp. The target must be a new drill name, never the source instance name.

```bash
AIOPS_SOURCE_DB=aiops-example-org-pg
AIOPS_COPY_DB=aiops-example-org-pg-drill
aws rds describe-db-instances --db-instance-identifier "$AIOPS_SOURCE_DB" \
  --query 'DBInstances[0].LatestRestorableTime'
```

Read [Capture a defensible source baseline](/help/ops-backup-restore-and-dr.md#capture-a-defensible-source-baseline) and prepare its program before proceeding. Record the timestamp and lag, then capture `source-counts.json` immediately after selecting T and **before the restore request**, with the quiet-window evidence described below. Then create a **single-AZ verification copy**, as the drills did; this is not the Multi-AZ production-cutover recipe.

```bash
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier "$AIOPS_SOURCE_DB" \
  --target-db-instance-identifier "$AIOPS_COPY_DB" \
  --restore-time "$AIOPS_RESTORE_TIME" --db-instance-class db.t4g.medium \
  --no-multi-az --db-subnet-group-name "$AIOPS_DB_SUBNET_GROUP" \
  --vpc-security-group-ids "$AIOPS_DB_SECURITY_GROUP" \
  --no-publicly-accessible --enable-iam-database-authentication \
  --no-deletion-protection
aws rds wait db-instance-available --db-instance-identifier "$AIOPS_COPY_DB"
aws rds modify-db-instance --db-instance-identifier "$AIOPS_COPY_DB" \
  --manage-master-user-password --apply-immediately
```

PostgreSQL PITR rejects `--manage-master-user-password` on the restore call. The separate modify call gives the copy its own managed password; do not read or reuse the source's master password. Confirm the copy's encryption, parameter group and private network settings against the source before use.

### Capture a defensible source baseline {#capture-a-defensible-source-baseline}

**`WHERE timestamp <= T` is not time travel.** A source read made after T cannot reconstruct rows deleted since T, changed timestamps, or the historic tenant population. Do not claim that current source counts equal a historic restore point merely because they are filtered by T.

Before selecting T, agree and record a quiet interval with the operations owner. It must cover T through source-baseline capture, with no deletion, timestamp edit, tenant change or other change to the counted population. Record how writers were controlled and the evidence confirming this; an operator's attestation is not automatically proved by the verifier. Pausing agents alone does not create a quiet interval: intake and other writes can continue. Allow existing audit work to seal before that interval so the set of chain heads is established. Keep the source operating on its original database throughout; this is a read-only comparison, not a service cutover.

Alternatively use an independently retained, approved baseline whose counts and chain membership describe exactly T. **Unknown or changed baseline = inconclusive**, not a backup failure or a successful restore check. Select another approved window or obtain the required baseline before acceptance.

The program below adapts the recorded drill's five SELECT queries and full `verify_chain` replay. Its endpoint, baseline and exit guards are additional documentation safeguards, checked locally without database calls. Save it as `dr-verify.py` on the operator workstation. It does not run application startup, migrations, a grant or a write query. The source mode uses the live pod's existing IAM login; the copy mode uses only the copy's new credential.

```python
import asyncio
import json
import os
import sys
from datetime import datetime, timezone
from urllib.parse import urlsplit

Q = {
    "auditevent": "select count(*) from auditevent where at <= :t",
    "incident": "select count(*) from incident where opened_at <= :t",
    "investigation": "select count(*) from investigation where started_at <= :t",
    "changerecord": "select count(*) from changerecord where created_at <= :t",
    "tenant": "select count(*) from tenant",
}


def compare(source, copy):
    if source.get("baseline_confirmed") is not True:
        return 2, "INCONCLUSIVE: source baseline has not been validated for T"
    if source["restore_time"] != copy["restore_time"]:
        return 1, "FAIL: restore timestamps differ"
    if source["db"][0] != "aiops" or copy["db"][0] != "aiops":
        return 1, "FAIL: unexpected database"
    if source["endpoint"] == copy["endpoint"] or source["db"][1] == copy["db"][1]:
        return 1, "FAIL: source and copy are not distinct"
    if any(type(source[k]) is not int or type(copy[k]) is not int for k in Q):
        return 1, "FAIL: a count is missing or invalid"
    if any(source[k] != copy[k] for k in Q):
        return 1, "FAIL: count mismatch; investigate baseline validity before blaming backups"
    if set(source["chains"]) != set(copy["chains"]):
        return 1, "FAIL: chain membership mismatch"
    results = copy.get("verify", {})
    if not results or set(results) != set(copy["chains"]):
        return 1, "FAIL: no complete chain replay results"
    for chain, result in results.items():
        if not (result.get("ok") and result.get("head_ok")
                and result.get("first_bad_seq") is None
                and result.get("checked") == result.get("last_seq")
                and result.get("last_seq") == copy["chains"][chain]):
            return 1, "FAIL: chain replay or head failed"
    return 0, "PASS: validated baseline counts and every copy chain match"


async def read_database():
    from sqlalchemy import text
    from app.config import settings
    from app.db import SessionLocal
    from app.aiops.audit_chain import verify_chain

    mode = os.environ["DRILL_MODE"]
    if mode not in ("source", "copy"):
        raise ValueError("mode")
    stamp = datetime.fromisoformat(os.environ["DRILL_T"].replace("Z", "+00:00"))
    if stamp.tzinfo is None:
        raise ValueError("restore timestamp must include timezone")
    stamp = stamp.astimezone(timezone.utc)
    endpoint = urlsplit(settings.db_url).hostname
    if endpoint != os.environ["DRILL_EXPECTED_ENDPOINT"]:
        raise ValueError("unexpected endpoint")
    async with SessionLocal() as session:
        db = list((await session.execute(text(
            "select current_database(), host(inet_server_addr()), current_user"
        ))).one())
        if db[0] != "aiops" or not db[1]:
            raise ValueError("unexpected database/server")
        if mode == "copy" and (db[1] == os.environ["DRILL_SRC_ADDR"]
                               or db[2] != os.environ["DRILL_MASTER"]):
            raise ValueError("copy identity mismatch")
        out = {"db": db, "endpoint": endpoint, "restore_time": stamp.isoformat(),
               "baseline_confirmed": mode == "source"
               and os.environ.get("DRILL_BASELINE_CONFIRMED") == "yes"}
        for name, query in Q.items():
            out[name] = (await session.execute(
                text(query), {"t": stamp.replace(tzinfo=None)}
            )).scalar_one()
        out["chains"] = dict((await session.execute(text(
            "select chain, last_seq from auditchainhead"
        ))).all())
        if mode == "copy":
            out["verify"] = {}
            for chain in out["chains"]:
                result = await verify_chain(session, chain, window=None)
                out["verify"][chain] = {key: result.get(key) for key in (
                    "ok", "checked", "first_bad_seq", "head_ok", "last_seq", "unsealed"
                )}
            if not out["verify"] or any(not r["ok"] for r in out["verify"].values()):
                print(json.dumps(out))
                return 1
        print(json.dumps(out))
        return 0


def main():
    try:
        if len(sys.argv) == 3:
            with open(sys.argv[1]) as stream:
                source = json.load(stream)
            with open(sys.argv[2]) as stream:
                copy = json.load(stream)
            code, message = compare(source, copy)
            print(message)
            return code
        return asyncio.run(read_database())
    except Exception as error:
        # Never print a connection URL, password or driver exception text.
        print("FAIL: " + type(error).__name__, file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())
```

Use the same T for both reads. Set `AIOPS_SOURCE_ENDPOINT` to the source RDS endpoint from its instance details. Set `DRILL_BASELINE_CONFIRMED=yes` only after the quiet-window/retained-baseline check above; leaving it unset makes final comparison inconclusive. The following source read is the recorded `kubectl exec` method, with the new guards and a saved JSON result:

```bash
kubectl -n aiops exec -i deploy/aiops -c aiops -- \
  env DRILL_MODE=source DRILL_T="$AIOPS_RESTORE_TIME" \
  DRILL_EXPECTED_ENDPOINT="$AIOPS_SOURCE_ENDPOINT" \
  DRILL_BASELINE_CONFIRMED="${DRILL_BASELINE_CONFIRMED:-no}" \
  python - < dr-verify.py > source-counts.json
```

Keep the JSON and quiet-window evidence in the drill record. The source query uses its existing `aiops` IAM user and only SELECTs. It contains counts, server address and full opaque chain ids; treat it as operating evidence, not a public example. Do not use a current source JSON as a substitute for an invalid historical baseline.

### Run the copy verifier securely {#run-the-copy-verifier-securely}

Use the approved platform image: the 8 October receipt used **0.1.108**. Capture the actual image reference from the source Deployment, including registry/tag or digest; both reads use that application code. The copy must be reachable from the verification pod through the private subnet/security group route. Do not make the database public to run this check.

In the approved account/region and Kubernetes context, read **only the copy's** managed-secret ARN, require a distinct endpoint, and stream its URL-encoded credential into a short-lived Kubernetes Secret. Never enable shell tracing, print the URL, write the credential to a local file or use the source master credential. The JSON-to-Secret stream avoids putting the password in `kubectl --from-literal` process arguments.

```bash
set -euo pipefail
set +x
: "${AIOPS_SOURCE_ENDPOINT:?set the source RDS endpoint}"
: "${AIOPS_COPY_DB:?set the disposable copy instance name}"
COPY_ENDPOINT=$(aws rds describe-db-instances --db-instance-identifier "$AIOPS_COPY_DB" \
  --query 'DBInstances[0].Endpoint.Address' --output text)
test -n "$COPY_ENDPOINT" && test "$COPY_ENDPOINT" != "$AIOPS_SOURCE_ENDPOINT"
COPY_SECRET_ARN=$(aws rds describe-db-instances --db-instance-identifier "$AIOPS_COPY_DB" \
  --query 'DBInstances[0].MasterUserSecret.SecretArn' --output text)
COPY_MASTER=$(aws rds describe-db-instances --db-instance-identifier "$AIOPS_COPY_DB" \
  --query 'DBInstances[0].MasterUsername' --output text)
VERIFY_IMAGE=$(kubectl -n aiops get deploy aiops \
  -o jsonpath='{.spec.template.spec.containers[0].image}')
SOURCE_ADDR=$(python3 -c 'import json; print(json.load(open("source-counts.json"))["db"][1])')
aws secretsmanager get-secret-value --secret-id "$COPY_SECRET_ARN" \
  --query SecretString --output text | COPY_ENDPOINT="$COPY_ENDPOINT" python3 -c '
import json, os, sys
from urllib.parse import quote
d = json.load(sys.stdin)
url = "postgresql+asyncpg://%s:%s@%s:5432/aiops" % (
    quote(d["username"], safe=""), quote(d["password"], safe=""), os.environ["COPY_ENDPOINT"])
json.dump({"apiVersion": "v1", "kind": "Secret", "metadata": {
    "name": "dr-drill-db", "namespace": "aiops"}, "type": "Opaque",
    "stringData": {"AIOPS_DB_URL": url}}, sys.stdout)
' | kubectl -n aiops create -f - >/dev/null
kubectl -n aiops create configmap dr-drill-program --from-file=dr-verify.py
```

Use fresh `dr-drill-*` names; stop if they belong to another operation. The following pod packages the recorded non-root, no-service-links copy read with the program mounted as a ConfigMap. Its metadata, mount and exit-result collection are documentation wrappers, not a newly measured live receipt. The Secret belongs only to the disposable copy. This deliberately omits the drill's unused conditional GRANT: a missing read privilege fails the check and needs a separately approved operator decision.

```bash
VERIFY_IMAGE="$VERIFY_IMAGE" COPY_ENDPOINT="$COPY_ENDPOINT" COPY_MASTER="$COPY_MASTER" \
  SOURCE_ADDR="$SOURCE_ADDR" DRILL_T="$AIOPS_RESTORE_TIME" python3 - <<'PY' | kubectl -n aiops create -f -
import json, os, sys
env = [{"name": key, "value": value} for key, value in {
    "DRILL_MODE": "copy", "DRILL_T": os.environ["DRILL_T"],
    "DRILL_EXPECTED_ENDPOINT": os.environ["COPY_ENDPOINT"],
    "DRILL_SRC_ADDR": os.environ["SOURCE_ADDR"], "DRILL_MASTER": os.environ["COPY_MASTER"],
    "AIOPS_DB_IAM_AUTH": "false",
}.items()]
env.append({"name": "AIOPS_DB_URL", "valueFrom": {"secretKeyRef": {
    "name": "dr-drill-db", "key": "AIOPS_DB_URL"}}})
json.dump({"apiVersion": "v1", "kind": "Pod", "metadata": {
    "name": "dr-drill-verify", "namespace": "aiops"}, "spec": {
    "restartPolicy": "Never", "enableServiceLinks": False,
    "automountServiceAccountToken": False,
    "securityContext": {"runAsNonRoot": True, "runAsUser": 1000},
    "containers": [{"name": "verify", "image": os.environ["VERIFY_IMAGE"],
        "command": ["python", "-c", "exec(compile(open('/verify/dr-verify.py').read(), '/verify/dr-verify.py', 'exec'))"], "env": env,
        "securityContext": {"allowPrivilegeEscalation": False, "capabilities": {"drop": ["ALL"]}},
        "volumeMounts": [{"name": "program", "mountPath": "/verify", "readOnly": True}]}],
    "volumes": [{"name": "program", "configMap": {"name": "dr-drill-program"}}],
}}, fp=sys.stdout)
PY
```

The `python -c` entrypoint keeps the image's `/srv` working directory on its module search path, as in the recorded drill, so the mounted program can import `app` without a package install or application startup.

Check termination and require exit code `0` before comparing; a container waiting/running is not completion. Capture logs only into the private evidence file and run the local, database-free comparison:

```bash
kubectl -n aiops get pod dr-drill-verify \
  -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
kubectl -n aiops logs dr-drill-verify > copy-counts.json
python3 dr-verify.py source-counts.json copy-counts.json
```

The program emits every full chain id from `auditchainhead` and its complete replay result; no eight-character truncation can merge two results. For every copy chain require `ok`, `head_ok`, `first_bad_seq: null` and `checked == last_seq == recorded copy head`. Keep reported unsealed rows visible in the receipt. The final comparison also requires the validated baseline's chain membership and five integer counts, same T/database and different source/copy endpoint **and** server address. It does not require the later source head sequence to equal the earlier copy's head.

Exit `0` means these bounded acceptance checks passed. Exit `1` is a query, identity, count or replay failure; exit `2` means baseline inconclusive. Only a completed pod with exit `0` **and** local comparison exit `0` can be recorded as a passed copy check. Investigate mismatches without assuming a post-T source mutation means a defective backup.

Delete the verifier resources even after an unsuccessful read. Preserve sanitized failure results first, then confirm these names belong to this drill:

```bash
kubectl -n aiops delete pod dr-drill-verify --ignore-not-found
kubectl -n aiops delete configmap dr-drill-program --ignore-not-found
kubectl -n aiops delete secret dr-drill-db --ignore-not-found
```

After recording successful verification, confirm `AIOPS_COPY_DB` is still the drill copy, then use the tested cleanup:

```bash
aws rds delete-db-instance --db-instance-identifier "$AIOPS_COPY_DB" \
  --skip-final-snapshot --delete-automated-backups
```

Record deletion completion and removal of its managed secret. Keep the source untouched. A failed verification is a failed drill; preserve the sanitized evidence and investigate before accepting backups as usable.

## Recover service and run the monthly check {#recover-service-and-run-the-monthly-check}

A real recovery additionally needs [Rollback, data restore](/help/ops-roll-back-a-release.md#restore-data-when-image-rollback-is-insufficient): database login authorization for the new instance, controlled name swap, control-plane restart and Terraform state reconciliation before another apply. Use the production settings in deployment runbook §4, not the single-AZ disposable-copy flags above. If restoring across a key rotation, deploy the retained key as `AIOPS_TENANT_SECRET_KEY_PREVIOUS` so startup can re-key the restored secrets. Verify connection decryption and execution separately, and replay every audit chain after restore.

Monthly, run the side-copy check in an approved window and retain source backup settings, restore point, lag, available/verified times, count comparison, chain outcomes, key-retention check and cleanup receipt. Include service cutover and connection-decryption results only when actually tested. The current cutover/state-import gap remains open.

Sources: [Availability](/help/ops-availability.md), [Deployment runbook §4 and §12.1](/help/ops-installation-runbook.md#4-rollback), [audit retention after restore](/help/ops-audit-retention.md#after-a-database-restore), and the two linked drill records.


## Next steps

- Next: [Platform monitoring and alerting](/help/ops-monitor-the-platform.md)
- [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt)
- [Documentation home](/help/)