# Thresholds and why

> Every number the platform works with — its value, why it has that value, what goes wrong lower or higher, and who changes it.

_Reference · Updated 2026-10-28 · Describes release 0.1.164_

The numbers the platform works with, why each has its value, what goes wrong if it is lower or higher, and who changes it. "Platform" means a platform administrator sets it with the environment variable shown, for every workspace. "Workspace" means a member with `members.manage` sets it, through the card named. The API is `PATCH /api/aiops/{tenant_id}/guardrails` for the cost, retention, scanner-sync, validation and **Change safety** fields, `PATCH /api/aiops/{tenant_id}/detection` for built-in detection, and `PATCH /api/aiops/{tenant_id}` with the settings key shown for the rest; each key's shape and bounds are in the [Settings reference](/help/settings-reference.md). "Fixed" means it is set in the code. Where the only reason is experience, the row says so.

## Grouping alerts into incidents {#thresholds-grouping}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Grouping horizon: `aiops_storm_group_window_s` 300 s + `aiops_storm_spread_window_s` 600 s | When a failing service's alert arrives, its incident takes in the open incidents of the failing services that depend on it, opened or reopened inside the two windows together (900 s), through the dependency edges allowed to fold. No count is involved: unrelated services stay separate incidents however many alert at once, except for a shared alert name (the row below). | The group window is how long an outage usually takes to consolidate on its root; the spread window lets it still reach callers afterwards. Experience, no measured basis. | Lower: a late caller opens its own incident. Higher: an older, unrelated incident on a caller is taken in. | Workspace: both keys (whole seconds, at most 3600 and 7200). A stored `aiops_storm_threshold` is ignored. |
| Grouping wait: `aiops_grouping_wait_s` 300 s; never for a P1 | An incident on a service that depends, over dependencies allowed to fold, on another service keeps its analysis back for the wait, so that service's alert arriving inside it takes the incident in; the analysis starts when the wait ends, or at **Analyse**. A potential root, an unrelated service and a concern a member raises never wait. A P1 (a `critical` alert always is) is not held, and a waiting incident starts at once when a merge or a joining `critical` alert makes it a P1, a member's concern joins it, or it takes in an incident under analysis. A dependency whose incident is already open is not waited for. | The window an outage consolidates in, from experience: in a live case two VMs failed 52 s apart and the root's alarm came 88 s after its caller's, 1 s after the caller's analysis escalated — an escalated incident is never regrouped. A P1 is not held because the worst incidents are analysed at once: five minutes of delay on a P1 costs more than the second analysis a later merge stops. | Lower: the root's alert arrives after the caller's analysis escalated it, and one outage gets two incidents and two analyses. Higher: a caller whose dependency never fails waits longer for its analysis, and an alert that clears inside the wait is not analysed (when it comes back, its analysis starts at once). | Workspace: `aiops_grouping_wait_s` (whole seconds, at most 600, `0` = no wait); the group window `aiops_storm_group_window_s` of the horizon above neither sets nor turns off the wait. Fixed: the P1 rule. |
| Grouping agent: `aiops_storm_agent` on; `AIOPS_STORM_AGENT_MIN_ALERTS` 5; `AIOPS_STORM_AGENT_INTERVAL_S` 300 s; 2 steps; 15 min | One short agent turn, with no connection reads, groups a related set of open incidents by root cause over every recorded edge origin, moves the victims to the root and merges drained incidents. A group earns a turn when it holds two or more incidents, at least 5 live alerts, and an incident no turn has seen in 15 minutes. | Experience: below 5 alerts a person reads them faster; 300 s lets more of the outage arrive between turns; 2 steps keep the turn cheap; 15 minutes keeps one group from being re-read every minute. | Lower: more turns on small groups. Higher: related incidents stay apart longer. | Workspace: `aiops_storm_agent`. Platform: the two numbers. Fixed: the steps and the 15 minutes. |
| Fold candidates: `AIOPS_FOLD_WINDOW_MIN` 30 min | Open incidents with an alert in the last 30 minutes are the ones a new alert may join through a declared dependency, a shared change or a shared alert name. | Experience: long enough for a failure to reach neighbouring services, short enough that an old incident does not collect unrelated alerts. | Lower: a late downstream alert opens its own incident. Higher: later, unrelated trouble joins an old incident. | Platform. Which dependency origins may fold: workspace `aiops_topology_fold_origins` (default `["declared"]`). |
| Shared change: `AIOPS_FOLD_CHANGE_WINDOW_S` 1800 s | A new alert joins an open incident pinned to the same change, made on the same service or resource within 1800 s before the error started. | Experience: a change usually shows its effect within minutes. | Lower: two incidents from one change stay apart. Higher: an older change is blamed. | Platform. |
| Shared alert name: `aiops_fold_shared_cause_threshold` 3 services | A new alert joins when its alert name is on 3 or more different services, counting its own service, among the open incidents above. Does not use the dependency map. | Experience, for one cause with one symptom across many services. | Lower: two services that share an alert name by chance are joined. Higher: a wide outage opens one incident per service. | Workspace (`0` = off). |
| Merge suggestions: `AIOPS_CORRELATOR_MIN_OPEN` 4 incidents in `AIOPS_CORRELATOR_WINDOW_S` 900 s; one call per `AIOPS_CORRELATOR_INTERVAL_S` 600 s | One light agent call suggests merges for ungrouped incidents; a person accepts or rejects each. Skipped while a budget or the kill switch blocks spend. | Experience: around 4 open incidents in 15 minutes a person loses the overview; 600 s keeps it to 6 calls an hour at most. | Lower: more calls and suggestions. Higher: related incidents are pointed out later. | Platform (`0` on the interval or the minimum = off). |

## Triage {#thresholds-triage}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Change harvest: `aiops_change_harvest_interval_s` 900 s | CloudTrail and Azure Activity Log writes into the change record. When an incident opens, the last 30 minutes are also read at once and again 5 minutes later; the first analysis waits up to 20 s for that read. | The read at incident open means the cause search never waits for the cycle; the second read is there because audit logs arrive after the write. While the interval stays at 30 minutes or less, it only sets how fresh the Changes page and change-triggered compliance checks are: the read at incident open covers the last 30 minutes, the cycle everything older. The CloudTrail lookup call is rate-limited by AWS. | Lower: more audit-log calls. Higher: the Changes page and change-triggered checks lag; above 30 minutes, a change made between 30 minutes and one cycle before the incident opened can be missing from the first analysis. | Workspace (`0` = off, which also stops the read at incident open). |
| Look-back cap: `aiops_lkg_lookback_h` 72 h | The furthest the change ranking reaches before the error started. The window usually starts later, at the last point with no error recorded since (the previous error's end, the alarm's last return to OK, or the last verified fix); with none it stops at the cap and says "not proven healthy". | Under a 99 % availability target (about 7 hours of downtime a month) a P1 should not stay open more than a few hours, so three days reaches back to a time the service worked and holds its last known good configuration. | Lower: a change before a long quiet period is missed. Higher: more old changes ranked; the relevant ones are harder to see. | Workspace (1–168). |
| Suspect window: `aiops_lkg_suspect_h` 2 h | A related change (for example the same resource, or a service up to two hops upstream) made in the 2 hours before the error is *suspect*; older related changes are *possible*. | Experience: most breaking changes break within minutes to an hour; 2 hours also covers a slow rollout. | Lower: a slow-burning change is not flagged. Higher: more changes flagged, the flag means less. | Workspace (1–24). |
| Proven good: `AIOPS_LKG_PROVEN_GOOD_H` 24 h | A change is *proven good* only when the alarm's own history shows OK on real data from before it until 24 h after it; such changes rank last. | One day includes the daily traffic peak. Experience. | Lower: a change that breaks only at the daily peak can be cleared. Higher: fewer changes can be cleared. | Platform (1–720). |
| Ranked changes in the prompt: `AIOPS_LKG_TOP_N` 5 | Only the top 5 ranked changes reach the analysis; the rest are counted. | Prompt size and focus. Experience. | Lower: the cause may be cut off. Higher: a longer, costlier prompt. | Platform (1–20). |
| Known-issue window: `aiops_triage_related_window_h` 24 h | Incidents on services one hop away count as known issues when open, or when resolved within 24 hours. | A neighbour fixed yesterday is still useful context; older ones are mostly noise. Experience. | Lower: this morning's neighbour fix drops out. Higher: old, unrelated incidents crowd the list. | Workspace (1 or more). |
| Resources of this service: 12 lines, 2,000 characters; 20 resources from 5 starting points | The resources of the alerting service in the estate model, walked from the incident's service and the resources its alerts name, each with the telemetry the workspace's connections read it by, reach the analysis before its first turn; the rest are counted with a pointer to **Service group**. | About 500 tokens, the size of the other context blocks (2,900 characters of change candidates, 900 of prior resolution). The 20 and 5 are technical limits that bound the walk on the control plane. | Lower: resources the analysis needs are cut. Higher: a longer prompt for every analysis. | Fixed. Off with the automated triage switch. |
| Known-issue fast path: 95 % and 3 steps; `aiops_known_issue_fast_path` on | An alert-started analysis rated at least 95 % (today only 100 reaches it) runs capped at 3 steps: the resources the fix touched, their one-hop neighbours and the current alert. The changes since the fix and the cloud health were already read by code. Never for a P1 or a Re-analyse. | 95 is the example from the customer review that asked for it. A full analysis takes up to 40 steps (`AIOPS_MAX_STEPS`) and a fix check 6; 3 is enough to confirm the three things that could make it another incident. On by default because the customer asked for it; a verified last-known-good record is needed, so it is rare. | Lower threshold: a match with a recorded change since the fix would be shortened. More steps: the shortcut costs more. | Workspace: **Automated triage**, *Shorten the analysis of a known issue*. Fixed: the 95 and the 3. |
| Known-issue relevancy: 100, 85, 70, 50, 0 % | The level on *Compare against known issues*: 100 = the same key resolved in the last 30 days (`AIOPS_PROBLEM_WINDOW_DAYS`) by a fix with a verified last-known-good record, the same alert names, no change recorded since the fix, no open provider issue, no reopen and no noise close on the newest same-key incident; 85 = that record, but one of the rest differs; 70 = a resolved same-key incident without such a record, or a reopen; 50 = an incident open one hop away; 0 = none. Cloud health *not available* never lowers it. | An order, not a probability: each level needs everything the one below has plus a stronger proof, all read by code. With `aiops_prior_resolution` off no prior incident is compared, so only a reopen (70) or an open neighbour (50) counts. | — | Fixed. |
| Ticket question: once per ticket; 10 service names; 60 s; 1 hour; 20 an hour | A ticket held for missing fields gets one note, 60 s after it arrives, listing the fields still missing, where its system takes each and the values accepted — for the service, at most 10 names from the dependency map and the estate model, then *…*. Only for a ticket created within the hour before it arrived, and at most 20 per workspace an hour; the rest wait unasked. A question whose turn never started is sent by the five-minute ticket sweep after 600 s; one cut off mid-call or refused by the ticket system is not sent again. | Once: a second note on someone's ticket is noise, and the update that names the fields is what moves it on. 10 names keep the note readable. 60 s lets an automation that fills a field at creation run first. The hour and the 20 keep a bulk edit or a mass import from putting a note on every ticket. | Shorter wait: a note asks for a field an automation was about to fill. Longer window or a higher cap: notes on old tickets or a burst of notes. | Fixed. Which fields: workspace `ticket_required_fields`. |
| Triage read cache: 600 s | An incident's cloud-health and ticket reads are reused by its next analysis within 10 minutes. | Avoids reading the provider's health and the ticket system twice for back-to-back runs. Experience. | Lower: duplicate reads. Higher: a provider event posted after the first read can be missed. | Fixed. |

## Validation after a fix {#thresholds-validation}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Settle time: 30 s | A typed cloud fix (security group or NSG rule, S3 Block Public Access, the IAM account password policy, blob public access, IMDSv2) is read back by code 30 s after the write. | EC2 and Azure Resource Manager reads are eventually consistent: a rule revoked a second ago can still be listed. Experience. | Lower: a good fix reads as failed. Higher: a later verdict. | Fixed. The check: **Validation checks (L6)**, `verify_typed_check`. |
| Alarm grace: 1800 s or more | With `aiops_verify_alert_rung` on (off by default), a CloudWatch alarm that reports its own recovery and is still in `ALARM` at the source after the grace (from the occurrence start, a close, an execution, a merge or a deployment) is `still_present`, with no agent run. The grace is the longer of 1800 s and the alarm's evaluation window plus 300 s. | An alarm needs its own evaluation periods and CloudWatch's delay to return to OK. Experience. | Lower: a fix judged failed while the alarm catches up. Higher: a failed fix found later. | Fixed. The rung: **Validation checks (L6)**, `verify_alert_rung`. |
| Validation rounds: `aiops_validation_rounds` 3, `aiops_validation_interval_s` 300 s | A change a **Post-change validation** rule watches (or, with `aiops_validation_derived` on, the signals its validation plan names) is read 3 times, one round every 300 s, the first once the write has settled; every round failed fails the fix, every round clean with the alerts cleared at their source validates it, a round 1 that alone read nothing is read again in one added round (the row below), anything else goes to the read-only verify. | The customer review asked for several reads 5 minutes apart and three were agreed: 15 minutes watched, and one bad reading alone never fails a fix. | Lower: one bad minute decides, or a slow recovery is called failed. Higher: a later verdict and a longer **Validating** state. | Workspace: **Change safety** → **Post-change validation** (*Rounds* 1–6, *Seconds apart* 60–1800). |
| Added round: 1 | When round 1 read nothing (not observed) and every later round was clean, the window reads one more round in its place before the read-only verify takes over; the verdict counts the rounds after round 1 and says *after 1 added round (round 1 read no data)*. Once per window. | The first round opens on the slowest signal's period 30 s after the write, often before a started instance has published a datapoint: live on 25 October a start at 01:04:28 left round 1 (01:05–01:10) without data although the fix worked, rounds 2 and 3 were clean, and the fix then waited 83 minutes for the verify. One round is the smallest wait that still gives the configured number of rounds with data. | 0: such a fix waits for the read-only verify. More: a fix whose signals stay silent reaches a person later. | Fixed. |
| Reachability analyses: `aiops_reachability_max_per_change` 10 a change; 60 s each; sweep after 1 h | With `aiops_reachability_checks` on (off by default), one analysis per neighbour with an EC2 network interface, once per window (a *not observed* one again); past the cap, or unfinished after 60 s, the neighbour reads *not observed*. While on, a sweep every 15 minutes removes `aiops:change` paths older than an hour. | About USD 0.10 an analysis on the customer's AWS bill: a plan lists at most 9 neighbours, so 10 is one each plus a retry (about USD 1 a change). 60 s keeps a round short (AWS states no completion time: experience); an hour outlasts any analysis in use. | Lower cap: more neighbours not observed. Higher: more spend. | Workspace: both keys (the cap 0–60). Fixed: 60 s, the hour, 15 minutes. |
| Hold after a start, reboot or restart: 600 s | For 10 minutes after a fix's start, reboot or restart step an automatic verify writes a $0 held row instead of starting the agent. A person's **Verify** still runs. | EC2 status checks report *initializing* and services need their readiness probes for minutes after a start: an agent checking sooner reads an instance that has not finished booting and answers *uncertain*, at a cost. Experience. | Lower: paid checks of a machine still booting. Higher: a failed restart is noticed later. | Fixed. |
| Verify run steps: `AIOPS_RECHECK_MAX_STEPS` 6 | The agent run that checks a fix when code could not decide. | A verify reads a few named signals. Experience. | Lower: more `uncertain` answers. Higher: more spend per check. | Platform. |
| Check order: `AIOPS_INCIDENT_RECHECK_BATCH` 5 a tick; a remediating incident first | The scheduled check takes 5 incidents a tick, one workspace's turn after another's, the longest unchecked first — except that an incident whose fix ran (*Remediating*) goes before the other incidents of its workspace. | That fix is only validated by this check: by age alone it waited behind every open scanner finding of its workspace (live on 25 October: 147 open findings at 5 a tick, 83 minutes). 5 is experience. | Lower: every check waits longer. Higher: more checks start at once. | Platform: the batch. Fixed: the order. |
| Verify attempts: `AIOPS_VERIFY_MAX_ATTEMPTS` 10 | After 10 post-fix checks that ran without a clear answer, the incident goes to a person; at the `AIOPS_INCIDENT_RECHECK_INTERVAL_S` 900 s cycle that is about 2.5 hours. A person's **Verify now** still runs after that. | Stops a check that can never conclude from polling forever. Experience. | Lower: slow-to-clear fixes reach a person early. Higher: an endless check costs more. | Platform. |
| Validation plan: 10 items, at most 5 of them changed resources | The change request's validation plan lists what the validation after the fix reads: each changed resource (at most 5) with its read and what it must show, the plan's own success check, then the services one hop away from the changed service, callers and callees in turn, until the plan holds 10 items. Only what the plan lists is read, and the approval covers that list. | The customer review set the scope at one hop of neighbours; 10 keeps each round's reads, and the list a person approves, short. Experience. | Lower: neighbours drop off the plan, and a break next door is not seen. Higher: more reads per round and a longer list to approve. | Fixed. |
| Close without a verdict: 24 h | A released change still *verifying* whose verdict has been none, or only `uncertain`, for 24 hours (from its first `uncertain` since the write, or from the write when no check answered) can be closed by a person with `approvals.decide`, with a reason. It becomes failed with the verdict `uncertain`; nothing rolls back on the close, and a later verdict still decides. | A day leaves room for the checks after a fix and for a scanner that reports once a day. Experience. | Lower: a change is closed before a slow verdict arrives. Higher: a change no check can answer stays open longer. | Fixed. |

## Recurrence and history {#thresholds-history}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Problem record: `AIOPS_PROBLEM_THRESHOLD` 3 within `AIOPS_PROBLEM_WINDOW_DAYS` 30 days; `AIOPS_PROBLEM_AUTOMATION_N` 3 | The 3rd same-key incident in 30 days opens or joins a problem record; a fix verified on 3 linked incidents is an automation candidate. The 30 days also bound the recurring tag and how old a verified fix may be for an automatic API fix to repeat it. | Experience: one repeat can be chance; a third within a month is a pattern. | Lower: one-offs become problems. Higher: recurring issues are flagged later. | Platform. |
| Reopen window: `aiops_reopen_window_h` 0 (off) | As in [Intake rules](/help/settings-reference.md#intake-rules); noise closes never reopen. Off, each occurrence is its own incident. | One incident per occurrence keeps its own analysis, cost and verdict; recurrence is counted by the problem record. | Higher: a new cause on the same service lands in an old incident's history. | Workspace: **Intake rules** (Alert intake & tokens), *Reopen window (hours)* (0–8760). |
| Retention: `AIOPS_RETENTION_DAYS_DEFAULT` 365 days | How long a closed record outlives its close; removal works in whole months. Its per-record costs go with it; the monthly billing totals stay. | One year keeps a full yearly review and a year of per-record costs visible. Experience; a workspace with a legal retention period sets its own. | Lower: older evidence is gone when asked for. Higher: more storage, records kept longer than needed. | Workspace: **Data retention**, `retention_days` (30–3650). |

## Scanner pull and spend {#thresholds-scanner-and-spend}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Scanner pull: `aiops_scanner_sync_interval_s` 900 s | Scheduled pull of Security Hub and Defender for Cloud findings through the read connections; Amazon Inspector findings too, unless the AWS connection's *Pull Inspector vulnerability findings (CVEs)* (Advanced connection settings) is `false`. Pull and webhook share deduplication. | On, because the cloud's own scanners are the detection layer: a finding reaches the platform within one cycle. Each HIGH or CRITICAL finding, CVEs included, arrives as P1/P2 and, while **Analyse P1/P2 findings automatically** is on (the default), starts one paid analysis. | Shorter interval: more Security Hub calls (throttled calls are retried). Longer: findings arrive later. | Workspace: **Scanner sync**, `scanner_sync_interval_s` (`0` = off, or 300–86400). The Inspector field: the AWS connection. |
| Per-run cap: `max_cost_usd`, platform `AIOPS_MAX_COST_USD_DEFAULT` 0 (uncapped) | Checked before each step. At 70 % of the cap the remaining steps move one model tier down (when tiers are set); at the cap the run stops without a summary and the incident escalates. | Measured in the test lab (August 2026): a run on the heaviest model used cost USD 0.39 at the median and USD 1.36 at the 90th percentile (all run types; upper bounds, as cached prompt reads were billed at the full input price). A cap several times the 90th percentile (for example USD 10) stops a looping run, not a hard incident. | Lower: hard incidents stop before their summary. Higher: a looping run costs more. | Workspace: **Cost cap & PII patterns**, `max_cost_usd`. Platform: the default. |
| Daily budget: `daily_cost_limit_usd`, platform `AIOPS_DAILY_COST_LIMIT_USD_DEFAULT` 0 (off) | Checked when a run starts; once the day's spend (UTC) reaches it, no new analysis starts and incidents wait for a person. A run already started finishes, so a day can end above the limit by up to one run cap per run in flight. | Off by default: the right limit depends on the workspace's incident volume. Size it from the expected incidents a day times the cost per analysis. | Lower: analyses stop earlier on a busy day. Higher: a burst costs more before anyone looks. | Workspace: **AI budgets & kill switch**, `daily_cost_limit_usd`. |

## Findings, patching and backup {#thresholds-findings-and-backup}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Raise from context: one tier up (at most L3) on an internet-facing or sensitive-data resource; CVSS 9 or more on an internet-facing resource = critical | A scanner finding on an internet-facing resource, or on one that holds sensitive data, has its fix tier raised one step; a CVE with a CVSS score of 9 or more on an internet-facing resource is also raised to critical, which opens at P1. A fix the raise lands at L3 waits for a person. Raise only: nothing is lowered. | 9.0 is where CVSS v3 starts its *Critical* band; on a resource reachable from the internet such a CVE can be exploited without a foothold inside. | Lower (for example 7): most HIGH CVEs on public resources become P1. Higher: none is raised. | Fixed. Exposure and sensitivity come from the resource's tags, the finding's details and the workspace's `risk_context`. |
| **Patch compliance** card: target 100 %; patch age `aiops_patch_within_days` 0 (no rule) | The share of AWS Systems Manager managed nodes Patch Manager reports compliant; EC2 instances Systems Manager does not manage are listed apart. With a patch-age rule, a node whose last patch operation ended longer ago counts as non-compliant too; without one the card shows 40 days as an example, with the nodes it would flag. Each non-compliant node also opens a Patching finding, unless Security Hub's SSM.2 already holds it. Read daily (`AIOPS_COMPLIANCE_REPORT_INTERVAL_S` 86400 s) and on **Refresh reports**. | A lower target would accept a known missing patch as normal. No patch age by default on purpose: the number is the customer's to name. | — | Fixed: the target and the 40-day example. Workspace: `aiops_patch_within_days` (0–3650), API only. |
| **Backup** card: AWS Backup jobs over the last 24 hours and the last 7 days; target 100 % | The share of finished AWS Backup jobs that completed, in each window, next to coverage (every EC2 instance in a backup plan, every RDS database with backup retention). Read daily, like the card above. | 24 hours shows last night's run; 7 days shows whether failures repeat, and includes a weekly plan. Our choice. | Shorter: one bad night decides the figure. Longer: a new failure hides behind older successes. | Fixed. |
| Pre-change backup: `aiops_backup_before_change` on; kept 7 days (`aiops_backup_retention_days`); waited for up to 30 minutes | Before a reboot-type step on an EC2 instance or an Azure VM, an AMI (taken without a reboot) or an OS-disk snapshot, waited for until it is available and tagged `aiops:retain-until` with the date it may go; no usable backup, no change. The platform never deletes a backup: your own clean-up removes it after that date. | 7 days covers the validation rounds, the automatic rollback check in the 24 hours after a failed fix, and a weekend before anyone looks. Experience. | Shorter: a fix that breaks later has no backup to restore from. Longer: the backups cost more until someone removes them. | Workspace: **Change safety**, *Back up a stateful target before a reboot-type change* and *Keep pre-change backups for (days)* (1–365). Fixed: the 30 minutes. |
| Recommended backup plan: daily at 18:00 UTC (02:00 in Singapore), recovery points kept 35 days | The plan a backup finding's commands create for an EC2 instance or RDS database with no recovery point. | 18:00 UTC is outside business hours in Singapore; 35 days follows AWS Backup's own plan template *Daily-35day-Retention*. Our proposal until your backup policy names its own. | Shorter retention: a problem found late may have no clean recovery point. Longer: higher backup storage cost. | Workspace: `aiops_backup_plan_schedule` (an AWS Backup `cron(...)` or `rate(...)` expression) and `aiops_backup_plan_retention_days` (0 = keep until deleted), API only. |
| Image reminder: `aiops_image_reminder_h` 24 h | While an image the platform [rebuilt under a new tag](/help/runbooks-and-automatic-fixes.md#image-rebuild) is not deployed (the scanner still reports the old digest, and the incident is open), targets that take `image_not_deployed` are reminded this often after the first message, which the validation sends. A rebuild more than 90 days old is no longer looked at. | Once a day reaches whoever deploys without a message on every scheduler tick. Experience. | Lower: a run of reminders on a slow deploy. Higher: a forgotten deploy keeps the old, vulnerable digest in use longer. | Workspace: `aiops_image_reminder_h` (0–720, `0` = the first message only), API only. Fixed: the 90 days. |

## Intake and change-record limits {#thresholds-intake-and-change-records}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Webhook body: `AIOPS_WEBHOOK_MAX_BODY_BYTES` 1 MiB; 8 MiB for Alertmanager, Grafana and the generic intake; 16 MiB for the Kubernetes audit webhook | A larger body is refused with `413` before it is read; each refusal is logged with the source and the size. | Before 0.1.149 any size the load balancer let through was read into memory before any check. 1 MiB fits the sources that send one alert or finding at a time. Alertmanager, Grafana and the generic intake send a whole alert group in one request, and a storm of a few thousand alerts passes 1 MiB; the Kubernetes API server batches its audit events. Our choice. | Lower: a large alert group is refused, and a sender does not retry a `413` (Alertmanager's `max_alerts` keeps a notification under the cap). Higher: one large body takes more memory on the control plane. | Platform (`0` = no cap). Fixed: the 8 and 16 MiB, which a larger platform value still raises. |
| CI/CD writes in Kubernetes: 50 objects per cluster and service account per clock hour | Writes by service accounts in the workspace's CI/CD namespaces are recorded as changes made by a service, one row per object; past 50 in the hour the rest are counted, not recorded one by one. Denied attempts and people are never capped. | A release of one application touches a handful of objects (its deployment, config maps, services), so it is recorded in full, while a bulk run, such as a sync of a whole cluster, cannot fill the change list. Experience. | Lower: a large release is cut short in the change record. Higher: one pipeline run fills the change list. | Workspace: the namespaces, `k8s_ci_namespaces` (default `flux-system`, `jenkins`, `gitlab-runner`, `tekton`; at most 20), API only. Fixed: the 50 and the hour. |
| Denied access: one SecOps incident per principal, account and hour | A cloud API call the provider refused is unauthorized access; nothing changed. While the SecOps agent is enabled, one P3 SecOps incident opens per principal and account in each hour, one alert per attempt inside it, with no analysis. | A script that retries a denied call would otherwise open an incident per attempt; one hour holds one burst, and a new incident in the next hour shows the attempts go on. Experience. | Shorter: more incidents for one burst. Longer: a new attempt hours later hides in an old incident. | Workspace: **Unmanaged changes**, *Report denied access attempts to SecOps* (`aiops_denied_access_to_secops`, on). Fixed: the hour. |

## Built-in detection {#thresholds-builtin-detection}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Poll: `AIOPS_BUILTIN_DETECTION_POLL_S` 300 s | One evaluation per workspace per poll. | Rules use 1-minute data (EC2: 5-minute); 5 minutes keeps the read cost low and still sees a breach within one poll. | Lower: higher CloudWatch read cost. Higher: later alerts. | Workspace: **Built-in detection** offers 60, 300 or 900 s; `poll_s` takes 60–3600 through the API. |
| Data delay: `AIOPS_BUILTIN_DETECTION_DELAY_S` 600 s | The newest datapoint read is 10 minutes old. | CloudWatch data lands late; Datadog's CloudWatch monitors use 900 s. | Lower: late data reads as missing and alerts flap. Higher: later alerts. | Platform. |
| Clear: 3 clean polls in a row | An alert ends after 3 polls with no breach; a failed read is not clean. | Azure's metric-alert resolve rule; stops close-and-reopen on one good datapoint. | Lower: flapping. Higher: incidents stay open longer after recovery. | Fixed. |
| Shadow: `AIOPS_BUILTIN_DETECTION_SHADOW_H` 24 h | After turning it on, rules are evaluated but send nothing. | One day, including the daily peak, shows which rules fire on normal behaviour. Experience. | Lower: noisy rules alert before they can be turned off. Higher: real problems wait longer. | Workspace: `shadow_hours` (0–168), through the API only. |
| Resource cap: `AIOPS_BUILTIN_DETECTION_MAX_RESOURCES` 300 | Resources per poll, round-robin across kinds. | Bounds the CloudWatch read bill; the card shows the estimate. | Lower: resources unwatched. Higher: higher cost and longer evaluations. | Workspace: **Built-in detection**, `max_resources` (1–5000). |
| Baseline (R26–R28): 95th percentile of 5-minute values over 7 days, read once a day, at least 288 values | What a latency or fault-rate rule compares against, per series. | A week holds the weekly cycle; the 95th percentile ignores one bad hour of it; 288 five-minute values are one day, so the daily peak is in it; daily, because a baseline moves slowly and 50 series × 7 days of 5-minute values fill one GetMetricData call (R26 reads two series per service, so two calls). AWS's TargetResponseTime alarm says to set the threshold from "the historical behavior of this metric". Ours. | Shorter window or lower percentile: normal peaks alert. Longer or higher: a slow regression hides in it. | Fixed. |
| Baseline margin: the larger of baseline × 1.5 and baseline + 100 ms (R27, R28); of 5 % and baseline + 5 points (R26) | The threshold above the baseline. | Two floors so neither a fast service's jitter (20 ms to 40 ms is no outage) nor a slow service's normal swing alerts; 100 ms is about the smallest delay a user notices. 5 % as R11 and R12. Ours. | Lower: alerts on normal variation. Higher: a real slowdown alerts later. | Fixed. |
| Baseline rule window: 3 of 3 × 5 min | R26–R28 and R30. | The baseline is made of 5-minute values, so the rule reads the same resolution; 15 minutes in a row is a degradation, not one slow garbage collection. A spike that recovers after 15 minutes still raises. Ours. | Shorter: one slow period alerts. Longer: later alerts. | Fixed. |
| Series per resource: 100 a poll | The pods and services read for one EKS cluster (R26, R27, R29, R30), shared round-robin across the rules, the rule with the fewest series first. | Bounds one cluster's read cost: 100 series every 5 minutes is about US$ 8.60 a month; round-robin, so a cluster's services never crowd out its one failed-node series. Ours. | Lower: big clusters' tails unwatched. Higher: cost grows with the cluster. | Fixed. |
| New rules in shadow: the workspace's shadow period (`shadow_hours`, 24 h) from the first poll the rule covers something | A rule added by a release after the workspace armed. | An upgrade never starts a rule firing unseen: one day of its would-have-fired hits shows whether it is noise before it opens anything. | — | Workspace: **Arm now** arms them at the next poll; `shadow_hours`. |
| Kubernetes waits: Pending 10 min (R33), not Ready 5 min (R34) | How long a pod may stay Pending, or a node not Ready, before it is an alert. | R33: cluster autoscaling adds a node and a large image pulls within a few minutes; 10 minutes is past both. Ours. R34: Kubernetes' own default — a not-ready node's pods are evicted after 300 s (the `node.kubernetes.io/not-ready` toleration). | Lower: a normal scale-up or a slow pull alerts. Higher: a stuck pod or a dead node alerts later. | Fixed. |
| Kubernetes read: 15 s a call, 90 s a cluster, 40 kubectl calls plus one per namespace listed on the connection | One cluster's read per poll (pods, nodes, namespaces), beside the CloudWatch reads; every cluster at the same time. | A cluster that does not answer gives up after its first call (15 s), and clusters are read side by side, so slow API servers never add up past the evaluation's 240 s timeout; the calls leave room for the namespace-by-namespace fallback of a namespace-scoped account, and a read the cap cuts short keeps every state. Experience. | Lower: big or namespace-scoped clusters fail their read. Higher: a dead cluster holds a worker thread longer. | Fixed. |
| State alerts open: 25 per connection (per account and region for scheduled maintenance, per account for provider health) | R31–R37. | A broken cluster or a host-reboot campaign is one line on the card, not hundreds of incidents; the rest open as those clear. Ours. | Lower: real failures wait behind open ones. Higher: one bad cluster floods the incident list. | Fixed. |
| Scheduled maintenance: read hourly, one clean read ends it, severity info (P4 unless criticality or blast radius raise it) | EC2 instance events and RDS pending maintenance (R35, R36). | AWS announces them days or weeks ahead, so an hour early is early enough and costs one call per account, region and service an hour; the event is a state AWS ends itself (completed, canceled), so one read without it is the end; P4 because nothing has failed yet — it is planned work on the resource. Ours, a proposal: whether it should be an incident or a notice only is still open. | Shorter: more calls, nothing earlier that matters. Longer: an event announced at short notice is seen later. | Fixed. |
| Provider health: every poll, cached 5 min, severity error (P2) | AWS's open issues for the services and regions of the in-scope estate (R37). | A provider outage is time-critical: read with every poll, but at most once per account every 5 minutes (the public dashboard once per set of regions and services; a role without `health:DescribeEvents` reads the public dashboard and is asked again an hour later), so a 1-minute poll adds no calls. P2 because the provider is degraded where the estate runs, not yet proven to hurt it; an incident only where the estate is exposed. Ours. | Shorter cache: more Health calls, nothing sooner that matters. Longer: an issue is seen later. | Fixed. |
| Rules R01–R37 | Thresholds and windows in [Built-in detection](/help/builtin-detection.md). | AWS *Recommended alarms (Classic)*, the ALB and NLB metric pages and the OpenSearch recommended alarms. Severity is the platform's choice. Ours (no vendor source gives the number): R09 20 ms; R10 and R13 the 1-minute period (R13 also 2 of 2); R11 and R12 5 %; every rate rule's floor of 10 requests or invocations a minute (R11, R12, R14, R16); R18 50 % of retention; R19 `> 0`. R14's 5 % follows Dynatrace's recommended AWS rule, where AWS alarms on errors above 0. R26–R30 (0.1.153): R26 5 %; R26, R27 and R28's floor of 10 requests a minute; R28 the p95 statistic; R29 `≥ 1` failed node (any node condition, in AWS's 5 of 5 × 1 min window for EKS Container Insights alarms); R30 `≥ 1` restart in each 5-minute period, because kubelet's crash-loop back-off caps at 5 minutes; R31's 3 restarts between back-offs; R33's 10 minutes. R31–R34 alert on the first read that sees the state (CrashLoopBackOff follows a failed restart). Every new rule is a warning. | Lower thresholds or shorter windows: alerts on normal behaviour. Higher or longer: real failures alert later or not at all. | Workspace: turn a rule off (`rules_off`), and **Warning rules start an analysis** (`warning_investigate`). Thresholds are fixed. |

## Synthetic checks {#thresholds-synthetic}

| Value · default | Where it applies | Why this value | Lower / higher | Who changes it |
|---|---|---|---|---|
| Interval: 300 s (60–600) | How often a check probes its target from the control plane. | The customer review asked for every 1 to 5 minutes; 5 minutes is the upper end, so a suggestion probes a customer endpoint at most 12 times an hour. | Lower: an outage is seen sooner, more requests to the endpoint. Higher (up to 600): an outage is seen later. | Workspace: **Every (seconds)** on each check. |
| Timeout: 10 s (1–30) | No answer within it is a failed run. | Experience: a page slower than 10 s is down for its users. | Lower: a slow endpoint reads as down. Higher: a hung endpoint holds the run longer. | Workspace: each check (API `timeout_s`). |
| Retry: one, 10 s later | A failed run is tried once more; only a failure that repeats counts. A spike that recovers within 10 s raises nothing. | Our proposal: one dropped connection or a single 502 during a deploy is not an outage. The customer asked that a spike which recovers still raises; that is open with them. | Lower (no retry): every blip opens an incident. Higher: a real outage is confirmed later. | Fixed. |
| Back to healthy: 2 passing runs in a row | A check that failed turns healthy, and its alert resolves, after two passes. | One pass after a failure can be the same flapping; two is the smallest run that is not. Experience. | Lower: close and reopen on a flapping endpoint. Higher: incidents stay open longer after recovery. | Fixed. |
| Severity: critical for your own checks, warning for a suggestion you keep | The severity of the alert a confirmed failure sends. | A check you added watches something you named as important; a kept suggestion was picked by the platform. | — | Workspace: **Severity** on each check. |
| Suggestions: `aiops_monitor_derive` on; at most 20; expected status 200-499 | Once a day the platform suggests a check for each internet-facing load balancer and API Gateway API in the workspace scope. A suggestion probes its endpoint (`GET /` every 300 s) in trial and opens nothing until a person keeps it. | The customer review asked for checks by default; any answer below 500 means the endpoint answered (`403` or `404` at `/` is normal). 20 bounds the requests a workspace sends without anyone asking. | Off: no new suggestion; one already listed keeps probing until it is discarded, unless its resource falls outside the scope (removed at the daily run). | Workspace: `aiops_monitor_derive`. Fixed: 20 and 200-499. |
| Checks per workspace: 100 | Every check, suggestions included. | Bounds the probes one workspace sends from the control plane. Experience. | — | Fixed. |


## Next steps

- Next: [Status and error reference](/help/status-and-error-reference.md)
- [All Reference pages](/help/reference/llms.txt)
- [Documentation home](/help/)