# Release notes 0.1.166

> 29 October 2026: The scale release: webhooks that no longer drain the database pool under load, a change capture that owes what it could not read instead of dropping it, and an estate history that…

_Release notes · Updated 2026-10-29 · Describes release 0.1.173_

## 0.1.166 — 29 October 2026 {#0-1-166-29-october-2026}

The scale release: webhooks that no longer drain the database pool under load, a change capture that owes what it could not read instead of dropping it, and an estate history that stops growing. It also clears the findings of an AWS account you disconnected and fixes one line of incident copy. Schema, additive and created at start: the table `loginfailure`. The chart adds a `dbPool` block with defaults and sets `AIOPS_API_REPLICAS`; no values change. New audit action `compliance.account_disconnected`. **One default changes behaviour and cannot be undone: the estate history is thinned** (opt-out below). Give every Amazon SNS HTTPS subscription a delivery policy before or with the upgrade (below).

### Webhooks under load {#webhooks-under-load}

- **A busy workspace answers `503` instead of failing.** Five workspaces sending about five webhooks a second drained the control plane's database pool: deliveries failed with `500` after 30 seconds and were lost. A delivery now authenticates, passes its rate limit and waits for its workspace's turn without a database connection. On the four routes whose senders retry a `5xx` on their own — `/webhook/alertmanager`, `/webhook/cloudwatch-sns`, `/webhook/azure-monitor` and `/webhook/securityhub` — it waits at most `AIOPS_INGEST_LOCK_TIMEOUT_S` (5 seconds); then the platform answers `503` with `Retry-After: 5` and stores nothing. The retry is deduplicated like any re-send. Every other route (Grafana, Datadog, Dynatrace, the ticket and change feeds, `/webhook/generic`) waits as long as it takes, as before, and gets a `503` only when no database connection frees within 30 seconds.
- **Retry advice for Amazon SNS.** Prometheus Alertmanager, Azure Monitor action groups and EventBridge API destinations retry a `503` for long enough. Amazon SNS retries an HTTPS delivery only 3 times, 20 seconds apart, unless the subscription has a longer delivery policy: give every SNS subscription that delivers CloudWatch alarms or Security Hub findings to the platform one (step 3 of *Set up alert sources* has the command), or a delivery that keeps meeting a busy workspace for a minute is lost. A sender of your own should retry a `503` with the same payload after the `Retry-After` delay.
- **Back-pressure per sender.** Each workspace's rate limit counts per replica (the limit divided by `AIOPS_API_REPLICAS`), and a payload larger than a replica's share passes on a full bucket instead of being refused for ever. A `503` hands its rate charge back, so the sender's retries do not run into a `429`.
- **A larger pool, sized from the chart:** each control-plane process opens up to 10 + 20 overflow connections (was 5 + 10), from the chart's new `dbPool` block. A webhook process uses at most half of them at once (`AIOPS_INGEST_ADMISSION`); in embedded mode, runs claiming and building their prompts use at most a quarter (`AIOPS_EMBEDDED_RUN_SLOTS`).
- **A disabled workspace or a rotated token is still refused at once.** The workspace's slug, token hash and settings are cached for `AIOPS_TENANT_CACHE_S` (30 seconds), and the platform reads the workspace again under its lock before it stores anything. A token minted or a workspace enabled on another replica is accepted on its first delivery.
- **Metrics:** `aiops_ingest_busy_total{reason}` counts each `503` (`turn`, `admission`, `lock`, `pool`). The example alert rules stop counting a `503` with `Retry-After` as an ingest failure and add `AiopsIngestBusy`.

### No lost change on a late CloudTrail delivery {#no-lost-change-on-a-late-cloudtrail-delivery}

- **Owed change windows.** A scheduled change pull reads at most 500 kept events (1,500 read) per connection; what did not fit used to be dropped. Now the connection owes the unread part of its window and reads it on the next pulls, oldest first, each region, EKS cluster or subscription taking its own share. The `change.harvested` audit row says what is owed and why, and `aiops_change_harvest_owed_seconds{tenant}` shows how far behind the oldest unread second is (alert `AiopsChangeCaptureBehind` above an hour).
- **Late deliveries are read.** CloudTrail and the Activity Log deliver some events up to 15 and 20 minutes late, so every span is read once more after 20 minutes. A long harvest interval (a daily one) still reads that late span on every pull. A throttled page is asked again after a backoff instead of failing the pull.
- **Gaps are counted, never silent.** A connection that comes back after more than a day without a scheduled pull reads its last day and counts the span before it `lost`; so does a window older than the 90 days the sources keep.

### Estate history {#estate-history}

- **The estate history stops growing — on by default, not reversible.** Each save of an estate item keeps the newest version, the active one and every version of the last 48 hours, then the first and the last of each day, and removes versions older than 30 days, at most 100 a save. A harvest keeps the newest 7 Turtle ABox snapshots per connection (never an x-ray scan's, one an attestation cites or a certification). Past 48 hours, last-known-good answers to the day rather than to the version. **Nothing is thinned while record retention archives (or is paused), or while the retention sweep is switched off (`AIOPS_RETENTION_SWEEP_ENABLED=false`):** thinning runs only when the sweep is on and its action is delete. A removed version comes back only from a database backup. To keep everything, set `AIOPS_ESTATE_HISTORY_DAYS=0` and `AIOPS_ESTATE_ABOX_KEEP=0` through `extraEnv` in the same upgrade.
- **Re-harvest after captured changes at most once an hour** per connection (was every minute); until then the touched resources stay marked stale and last-known-good reads them live.

### Background work {#background-work}

- **Detached pulls.** The change harvest, the scanner pull, the estate inventory and the synthetic-check suggestions run beside the scheduler tick, one at a time, so a pull that catches up no longer holds the tick back. The estate inventory takes that turn per connection, so a daily sweep over many connections lets the change harvest and the scanner pull run between two of them. A scanner pull lands its findings 200 at a time (`AIOPS_SCANNER_CHUNK`), so a webhook waits behind one chunk, not the whole pull. A pull stops between chunks when its replica is no longer the leader, and the expiry sweep of risk decisions waits for the workspace's ingest.
- **Login failures in the database.** The login lockout counts failures in the new `loginfailure` table (a hash of the email and the time, never the address), so it trips once whichever replica got the attempts and a restart keeps the count. Rows older than a day are removed. A lock in place before the upgrade does not carry over.
- **State that holds across replicas.** The SSO email-domain claim, the MCP connection receipt, a full audit-chain replay and the SMTP and guardrails saves are serialised in the database. The control plane still runs one replica.
- **A scanner re-stamp no longer rewrites the finding:** a re-sent finding whose only change is its update or observation time keeps its stored payload.

### Compliance and incidents {#compliance-and-incidents}

- **Findings of a disconnected account clear.** After a scanner pull whose every AWS read was whole, an open Security Hub finding of an account that a read connection once named (its connection disabled, or deleted) and no enabled one names any more, that the pull did not return and that was not seen for 48 hours is cleared as *account no longer connected* — never counted as fixed (audit `compliance.account_disconnected`). Before, it stayed open until Security Hub's 90-day retention. Nothing is cleared while an enabled AWS connection's account is unknown, and an account no connection ever named (its findings come by webhook) keeps its findings until that retention, as before.
- **The incident's plain line names an action.** An incident whose plan has no runnable step read *You: Not applicable — no runnable fix.*; it now reads *Next: run the manual steps yourself, then mark the incident resolved.*

### Review fixes {#review-fixes}

- The rule dictionary the automation in your account reads re-checks the workspace first: a workspace disabled since gets nothing.
- The other fixes from the release review are part of the items above: the daily-interval late span, the leader check on detached pulls, the first-delivery acceptance of a new token, no thinning while retention archives or its sweep is off, the rate charge handed back on a `503` (the change feed's too), the change feed re-reading the workspace before it records a second automation, and a webhook-fed account's findings kept open.


## Next steps

- Next: [Release notes 0.1.165](/help/release-notes-0-1-165.md)
- [All What's new pages](/help/what-s-new/llms.txt)
- [Documentation home](/help/)