Security operations

Maintain the vault copy of deployment secrets, review access, and keep a verified audit archive outside the database.

How-to · Updated 2026-10-26 · Describes release 0.1.164

Evidence reviewed 8 October 2026; operating baseline 0.1.130 (the 16 October 2026 handover build; see the release notes).

Maintain the vault copy of deployment secrets, review access, and keep a verified audit archive outside the database. The baseline is 0.1.130; secret-store rotation and audit Verify/Export are delivered, and so is operational-record retention (0.1.114). Audit retention is a separate policy and archive responsibility: nothing truncates the audit chain, and some hardening items remain open.

Secret inventory and rotation

For pipeline secrets, edit the onboarding secrets file, run customer-onboard.py ado <slug> --update-group, then queue an approved deployment with the live image/chart. The file wins over manual Library edits. Keep secret contents out of change tickets and logs.

Tenant secret-store key

These commands are quoted from deployment runbook §12.1. Set the installation slug on the release operator's secure workstation. The census runs in the approved EKS context and contains counts, key fingerprint and opaque ids, not credential values.

kubectl -n aiops exec deploy/aiops -- \
  python -m app.aiops.secrets rekey --dry-run > census.json

1. Start from a build supporting re-key that has already booted once. Check previous key is empty, vault the current file/key and record a clean census: remaining: 0, unreadable: []; note readable.

2. Rotate from that census; vault the new file again, then update the group:

   AIOPS_SLUG=example-org
   python3 scripts/customer-onboard.py rotate-key "$AIOPS_SLUG" --census census.json
   python3 scripts/customer-onboard.py ado "$AIOPS_SLUG" --update-group
   

3. Deploy the current image with both keys. The customer branch must contain the previous-key mapping. Startup compare-and-swap re-keys rows; each readable changed row gets secret.rekeyed, with a platform secrets.rekey_completed summary. Test connections.

4. Check the newest summary after the successful deploy started: new key_id, remaining: 0, unreadable: [] and expected readable count. Save that summary event as event.json from Ops → Audit log.

5. Retire only with that current successful report, update the group, deploy again and Test again:

   python3 scripts/customer-onboard.py retire-key "$AIOPS_SLUG" --rekey-report event.json
   python3 scripts/customer-onboard.py ado "$AIOPS_SLUG" --update-group
   

Stop if any row is unreadable or remains under the old key. Do not retire. Fix missing previous-key plumbing and deploy again; unreadable rows stay untouched and connection edits fail closed with 409. The report must be from after the latest successful deploy, not an older successful rotation.

Keep AIOPS_SECRET_KEY_RETIRED_<YYYYMMDD> in the vault until all pre-rotation snapshot/PITR windows expire; it is never copied to the pipeline group. A restore from that period needs it as the pod's AIOPS_TENANT_SECRET_KEY_PREVIOUS. To reverse a rotation swap the current and previous values in the source file and re-run the approved two-key deploy. Do not Helm-rollback across the rotation boundary; see Rollback. API-mode runners never receive either key; database-mode runners restart with changed key environment.

Access review

Weekly inspect changes, and monthly retain a signed-off review:

1. Ops → Users: active staff, superadmins and unused accounts. Workspace owner: Members & roles, custom permission sets and invited users.
2. Review IdP issuer/domain and group-to-role mappings against the current joiner/mover/leaver list. Do not treat a matching email domain as proof of domain ownership; that proof is not implemented.
3. Review active support grants and revoke completed or unnecessary grants. Staff have no automatic workspace-data bypass; use the expiring support grant for a named reason and revoke it after the work.
4. Review PAT scope/expiry, intake sender ownership, agent tokens, source estate IAM grants and execution consent. For templates from 0.1.109, also review control-plane and runner sts:AssumeRole/sts:TagSession access to arn:aws:iam::*:role/aiops-*, the estate roles and External IDs that trust them, and each workspace's staff-prepared pair at GET /api/aiops/admin/tenants/{id}/cloud-runtime. Revoke departed-user and unused-client access.
5. Use Audit log filters to evidence user, member, role, support and connection changes. Record reviewer, UTC period, exceptions, owner and due date. There is no built-in access-review report.

The bootstrap admin is deliberately re-enabled/re-promoted at startup from deployment configuration and audited. Removing privileges only in the UI does not disable this break-glass design. A password change ends the account's other browser sessions, and a staff reset, a reset link or a forced bootstrap password ends all of them; API tokens survive a password change, so revoke them separately. Disabling an account denies its sessions/tokens on the next request.

Verify and archive the audit trail

In the portal Audit log use Verify chain, and in Ops Audit log use the chain verification controls. Use a full replay after restore or suspected tampering. Recent rows can remain unsealed for about a minute; investigate a sustained backlog through the external monitoring rules.

Use Export for a review. For the monthly independent archive, follow the retention policy's cursor procedure through every page (up to 50,000 rows), until X-Truncated is false. Archive unfiltered sequence exports, not an action/date-filtered subset. Each export is itself audited. Verify every NDJSON file with the policy's tested offline command, using the exported filename:

python3 scripts/verify-audit-export.py audit-export.ndjson

Require OK and no gap: line. Link the first file back to the preceding archive's last sequence/hash and each later file to its predecessor. Store files plus manifest in customer write-once storage: chain id, hashes, first and last sequence, last row hash, returned chain head, export time and actor. The external manifest anchors the chain against a database attacker who could rewrite both rows and in-database head.

Online audit rows remain indefinitely: deletion or truncation of the audit chain is not delivered, and the retention sweep below never touches it. Archive retention is to be agreed with the customer; record the period, agreement date and decision maker. Do not run a database delete as retention.

Operational-record retention

Operational-record retention shipped in 0.1.114 and is distinct from audit retention. Review the live setting, its preview and its last sweep rather than inferring deletion from the age of a document.

1. Read the workspace window on Guardrails & cost → Data retention: the window in force and its source, and whether the workspace follows the platform default (Use platform default). The platform default is set on Platform defaults → Data retention default in the Ops console (30 to 3650 days; AIOPS_RETENTION_DAYS_DEFAULT, shipped 365).
2. Use Preview the next sweep before changing a window. It is a dry run with the UTC cutoff and the statements it would seal; a preview is not evidence of deletion.
3. Check that the sweep is enabled (AIOPS_RETENTION_SWEEP_ENABLED, shipped true), and read the recorded retention_last_sweep result with its counts or error. An attempt that did not complete says so and is retried a day later. A configured window without a completed result is not proof that anything was removed.
4. Review the retention.swept audit event when records were removed. Keep the cutoff, the counts and the sealed-month usage statement with the operating record: a past billing statement never shrinks because of a sweep.

Never removed: anything open or active, a record with a run still running or paused or with a run or validation in or after the cutoff's month, a cleared finding with a risk decision in force or awaiting sign-off, the audit trail, knowledge, skill proposals, attestations and the evidence they cite, posture snapshots, benchmark datasets and runs, Test Lab runs, certification artifacts and configuration. Do not perform ad hoc database deletes to approximate the sweep. Operating procedure: Release notes 0.1.114.

The record archive

With AIOPS_ARCHIVE_BUCKET set, the sweep archives before it removes (DESIGN.md §13.8). Review it alongside the steps above:

1. On Platform defaults → Data retention default, read the action (Archive, then remove or Remove), the archive storage line — the bucket, its region and the credential method, which must be container-role or assume-role-with-web-identity — and the workspaces whose last attempt failed. Paused means nothing is being removed: the archive was chosen or has run, and the bucket setting is gone.
2. For each retention.archived audit event, keep batch_id, month, held, data_sha256 and manifest_sha256 with the operating record. GET /api/aiops/{workspace}/archive/batches lists the batches with their state, counts and manifest SHA-256; the manifest object's SHA-256 must equal the audit row's.
3. Check the gauge aiops_archive_last_success_timestamp_seconds{tenant}: an archive workspace whose value is more than two days old is not being archived (the alert in platform monitoring).
4. training_data.exported audit events record every fine-tuning export (one archived month, members.manage).

Break-glass: erasing an archived record inside its retention period. The archive bucket's Object Lock (governance mode, 5 years by default) and its policies let only the control plane's role read or write objects, and that role cannot delete or shorten the retention. Erasure — a legal request, an offboarded customer — is an infrastructure change, never a console action:

1. Record the request, the records or the workspace prefix (tenants/<workspace id>/), the decision maker and the date.
2. Add a dedicated role to the archive unit's breakglass_role_arns through the infrastructure pipeline (reviewed like any infra change), with s3:BypassGovernanceRetention and s3:DeleteObject/s3:DeleteObjectVersion on that prefix only. The bucket policy denies the account root too; root could only rewrite that policy, so always use a break-glass role.
3. Delete every version of the objects with the governance bypass header, then list the prefix's versions to confirm nothing is left. A batch is one object per month: erasing one record means rewriting that month without it, which this release does not automate — erase the month, or the prefix.
4. Remove the role from breakglass_role_arns in the same change window and record the evidence. The ArchivedRecord index rows of erased batches are left as tombstones; their reads then answer 502.

Network destinations to review

Review the egress the installation actually needs and record the destination, its purpose, the requesting component and a successful test. Beyond the model gateway and the estate endpoints:

An egress list limited to the five categories of the original deployment can block a supported feature. Do not open unrestricted egress to make a test pass.

Accepted risks to review before production

Keep the full security review with the operating record. Its actionable risks include unenforced network policies, shared admin/public ingress, optional internet tools, telnet, in-database audit anchors, plaintext secrets without a key, unsigned images, unverified domain claims and non-reproducible base/kubectl builds. Prefer SSH, source-enforced read-only grants and explicit allowed egress. The dedicated deployment account compensates for the broad deploy-role policy; its permissions boundary remains open.

Sources: Deployment runbook §12–12.1, audit policy, People, access and audit, and Security review.

Next steps

• Next: Platform incident response
• All Install and operate the platform pages
• Documentation home