Thresholds and why

Every number the platform works with — its value, why it has that value, what goes wrong lower or higher, and who changes it.

Reference · Updated 2026-10-28 · Describes release 0.1.164

The numbers the platform works with, why each has its value, what goes wrong if it is lower or higher, and who changes it. "Platform" means a platform administrator sets it with the environment variable shown, for every workspace. "Workspace" means a member with members.manage sets it, through the card named. The API is PATCH /api/aiops/{tenant_id}/guardrails for the cost, retention, scanner-sync, validation and Change safety fields, PATCH /api/aiops/{tenant_id}/detection for built-in detection, and PATCH /api/aiops/{tenant_id} with the settings key shown for the rest; each key's shape and bounds are in the Settings reference. "Fixed" means it is set in the code. Where the only reason is experience, the row says so.

Grouping alerts into incidents

Value · default Where it applies Why this value Lower / higher Who changes it
Grouping horizon: aiops_storm_group_window_s 300 s + aiops_storm_spread_window_s 600 s When a failing service's alert arrives, its incident takes in the open incidents of the failing services that depend on it, opened or reopened inside the two windows together (900 s), through the dependency edges allowed to fold. No count is involved: unrelated services stay separate incidents however many alert at once, except for a shared alert name (the row below). The group window is how long an outage usually takes to consolidate on its root; the spread window lets it still reach callers afterwards. Experience, no measured basis. Lower: a late caller opens its own incident. Higher: an older, unrelated incident on a caller is taken in. Workspace: both keys (whole seconds, at most 3600 and 7200). A stored aiops_storm_threshold is ignored.
Grouping wait: aiops_grouping_wait_s 300 s; never for a P1 An incident on a service that depends, over dependencies allowed to fold, on another service keeps its analysis back for the wait, so that service's alert arriving inside it takes the incident in; the analysis starts when the wait ends, or at Analyse. A potential root, an unrelated service and a concern a member raises never wait. A P1 (a critical alert always is) is not held, and a waiting incident starts at once when a merge or a joining critical alert makes it a P1, a member's concern joins it, or it takes in an incident under analysis. A dependency whose incident is already open is not waited for. The window an outage consolidates in, from experience: in a live case two VMs failed 52 s apart and the root's alarm came 88 s after its caller's, 1 s after the caller's analysis escalated — an escalated incident is never regrouped. A P1 is not held because the worst incidents are analysed at once: five minutes of delay on a P1 costs more than the second analysis a later merge stops. Lower: the root's alert arrives after the caller's analysis escalated it, and one outage gets two incidents and two analyses. Higher: a caller whose dependency never fails waits longer for its analysis, and an alert that clears inside the wait is not analysed (when it comes back, its analysis starts at once). Workspace: aiops_grouping_wait_s (whole seconds, at most 600, 0 = no wait); the group window aiops_storm_group_window_s of the horizon above neither sets nor turns off the wait. Fixed: the P1 rule.
Grouping agent: aiops_storm_agent on; AIOPS_STORM_AGENT_MIN_ALERTS 5; AIOPS_STORM_AGENT_INTERVAL_S 300 s; 2 steps; 15 min One short agent turn, with no connection reads, groups a related set of open incidents by root cause over every recorded edge origin, moves the victims to the root and merges drained incidents. A group earns a turn when it holds two or more incidents, at least 5 live alerts, and an incident no turn has seen in 15 minutes. Experience: below 5 alerts a person reads them faster; 300 s lets more of the outage arrive between turns; 2 steps keep the turn cheap; 15 minutes keeps one group from being re-read every minute. Lower: more turns on small groups. Higher: related incidents stay apart longer. Workspace: aiops_storm_agent. Platform: the two numbers. Fixed: the steps and the 15 minutes.
Fold candidates: AIOPS_FOLD_WINDOW_MIN 30 min Open incidents with an alert in the last 30 minutes are the ones a new alert may join through a declared dependency, a shared change or a shared alert name. Experience: long enough for a failure to reach neighbouring services, short enough that an old incident does not collect unrelated alerts. Lower: a late downstream alert opens its own incident. Higher: later, unrelated trouble joins an old incident. Platform. Which dependency origins may fold: workspace aiops_topology_fold_origins (default ["declared"]).
Shared change: AIOPS_FOLD_CHANGE_WINDOW_S 1800 s A new alert joins an open incident pinned to the same change, made on the same service or resource within 1800 s before the error started. Experience: a change usually shows its effect within minutes. Lower: two incidents from one change stay apart. Higher: an older change is blamed. Platform.
Shared alert name: aiops_fold_shared_cause_threshold 3 services A new alert joins when its alert name is on 3 or more different services, counting its own service, among the open incidents above. Does not use the dependency map. Experience, for one cause with one symptom across many services. Lower: two services that share an alert name by chance are joined. Higher: a wide outage opens one incident per service. Workspace (0 = off).
Merge suggestions: AIOPS_CORRELATOR_MIN_OPEN 4 incidents in AIOPS_CORRELATOR_WINDOW_S 900 s; one call per AIOPS_CORRELATOR_INTERVAL_S 600 s One light agent call suggests merges for ungrouped incidents; a person accepts or rejects each. Skipped while a budget or the kill switch blocks spend. Experience: around 4 open incidents in 15 minutes a person loses the overview; 600 s keeps it to 6 calls an hour at most. Lower: more calls and suggestions. Higher: related incidents are pointed out later. Platform (0 on the interval or the minimum = off).

Triage

Value · default Where it applies Why this value Lower / higher Who changes it
Change harvest: aiops_change_harvest_interval_s 900 s CloudTrail and Azure Activity Log writes into the change record. When an incident opens, the last 30 minutes are also read at once and again 5 minutes later; the first analysis waits up to 20 s for that read. The read at incident open means the cause search never waits for the cycle; the second read is there because audit logs arrive after the write. While the interval stays at 30 minutes or less, it only sets how fresh the Changes page and change-triggered compliance checks are: the read at incident open covers the last 30 minutes, the cycle everything older. The CloudTrail lookup call is rate-limited by AWS. Lower: more audit-log calls. Higher: the Changes page and change-triggered checks lag; above 30 minutes, a change made between 30 minutes and one cycle before the incident opened can be missing from the first analysis. Workspace (0 = off, which also stops the read at incident open).
Look-back cap: aiops_lkg_lookback_h 72 h The furthest the change ranking reaches before the error started. The window usually starts later, at the last point with no error recorded since (the previous error's end, the alarm's last return to OK, or the last verified fix); with none it stops at the cap and says "not proven healthy". Under a 99 % availability target (about 7 hours of downtime a month) a P1 should not stay open more than a few hours, so three days reaches back to a time the service worked and holds its last known good configuration. Lower: a change before a long quiet period is missed. Higher: more old changes ranked; the relevant ones are harder to see. Workspace (1–168).
Suspect window: aiops_lkg_suspect_h 2 h A related change (for example the same resource, or a service up to two hops upstream) made in the 2 hours before the error is suspect; older related changes are possible. Experience: most breaking changes break within minutes to an hour; 2 hours also covers a slow rollout. Lower: a slow-burning change is not flagged. Higher: more changes flagged, the flag means less. Workspace (1–24).
Proven good: AIOPS_LKG_PROVEN_GOOD_H 24 h A change is proven good only when the alarm's own history shows OK on real data from before it until 24 h after it; such changes rank last. One day includes the daily traffic peak. Experience. Lower: a change that breaks only at the daily peak can be cleared. Higher: fewer changes can be cleared. Platform (1–720).
Ranked changes in the prompt: AIOPS_LKG_TOP_N 5 Only the top 5 ranked changes reach the analysis; the rest are counted. Prompt size and focus. Experience. Lower: the cause may be cut off. Higher: a longer, costlier prompt. Platform (1–20).
Known-issue window: aiops_triage_related_window_h 24 h Incidents on services one hop away count as known issues when open, or when resolved within 24 hours. A neighbour fixed yesterday is still useful context; older ones are mostly noise. Experience. Lower: this morning's neighbour fix drops out. Higher: old, unrelated incidents crowd the list. Workspace (1 or more).
Resources of this service: 12 lines, 2,000 characters; 20 resources from 5 starting points The resources of the alerting service in the estate model, walked from the incident's service and the resources its alerts name, each with the telemetry the workspace's connections read it by, reach the analysis before its first turn; the rest are counted with a pointer to Service group. About 500 tokens, the size of the other context blocks (2,900 characters of change candidates, 900 of prior resolution). The 20 and 5 are technical limits that bound the walk on the control plane. Lower: resources the analysis needs are cut. Higher: a longer prompt for every analysis. Fixed. Off with the automated triage switch.
Known-issue fast path: 95 % and 3 steps; aiops_known_issue_fast_path on An alert-started analysis rated at least 95 % (today only 100 reaches it) runs capped at 3 steps: the resources the fix touched, their one-hop neighbours and the current alert. The changes since the fix and the cloud health were already read by code. Never for a P1 or a Re-analyse. 95 is the example from the customer review that asked for it. A full analysis takes up to 40 steps (AIOPS_MAX_STEPS) and a fix check 6; 3 is enough to confirm the three things that could make it another incident. On by default because the customer asked for it; a verified last-known-good record is needed, so it is rare. Lower threshold: a match with a recorded change since the fix would be shortened. More steps: the shortcut costs more. Workspace: Automated triage, Shorten the analysis of a known issue. Fixed: the 95 and the 3.
Known-issue relevancy: 100, 85, 70, 50, 0 % The level on Compare against known issues: 100 = the same key resolved in the last 30 days (AIOPS_PROBLEM_WINDOW_DAYS) by a fix with a verified last-known-good record, the same alert names, no change recorded since the fix, no open provider issue, no reopen and no noise close on the newest same-key incident; 85 = that record, but one of the rest differs; 70 = a resolved same-key incident without such a record, or a reopen; 50 = an incident open one hop away; 0 = none. Cloud health not available never lowers it. An order, not a probability: each level needs everything the one below has plus a stronger proof, all read by code. With aiops_prior_resolution off no prior incident is compared, so only a reopen (70) or an open neighbour (50) counts. — Fixed.
Ticket question: once per ticket; 10 service names; 60 s; 1 hour; 20 an hour A ticket held for missing fields gets one note, 60 s after it arrives, listing the fields still missing, where its system takes each and the values accepted — for the service, at most 10 names from the dependency map and the estate model, then …. Only for a ticket created within the hour before it arrived, and at most 20 per workspace an hour; the rest wait unasked. A question whose turn never started is sent by the five-minute ticket sweep after 600 s; one cut off mid-call or refused by the ticket system is not sent again. Once: a second note on someone's ticket is noise, and the update that names the fields is what moves it on. 10 names keep the note readable. 60 s lets an automation that fills a field at creation run first. The hour and the 20 keep a bulk edit or a mass import from putting a note on every ticket. Shorter wait: a note asks for a field an automation was about to fill. Longer window or a higher cap: notes on old tickets or a burst of notes. Fixed. Which fields: workspace ticket_required_fields.
Triage read cache: 600 s An incident's cloud-health and ticket reads are reused by its next analysis within 10 minutes. Avoids reading the provider's health and the ticket system twice for back-to-back runs. Experience. Lower: duplicate reads. Higher: a provider event posted after the first read can be missed. Fixed.

Validation after a fix

Value · default Where it applies Why this value Lower / higher Who changes it
Settle time: 30 s A typed cloud fix (security group or NSG rule, S3 Block Public Access, the IAM account password policy, blob public access, IMDSv2) is read back by code 30 s after the write. EC2 and Azure Resource Manager reads are eventually consistent: a rule revoked a second ago can still be listed. Experience. Lower: a good fix reads as failed. Higher: a later verdict. Fixed. The check: Validation checks (L6), verify_typed_check.
Alarm grace: 1800 s or more With aiops_verify_alert_rung on (off by default), a CloudWatch alarm that reports its own recovery and is still in ALARM at the source after the grace (from the occurrence start, a close, an execution, a merge or a deployment) is still_present, with no agent run. The grace is the longer of 1800 s and the alarm's evaluation window plus 300 s. An alarm needs its own evaluation periods and CloudWatch's delay to return to OK. Experience. Lower: a fix judged failed while the alarm catches up. Higher: a failed fix found later. Fixed. The rung: Validation checks (L6), verify_alert_rung.
Validation rounds: aiops_validation_rounds 3, aiops_validation_interval_s 300 s A change a Post-change validation rule watches (or, with aiops_validation_derived on, the signals its validation plan names) is read 3 times, one round every 300 s, the first once the write has settled; every round failed fails the fix, every round clean with the alerts cleared at their source validates it, a round 1 that alone read nothing is read again in one added round (the row below), anything else goes to the read-only verify. The customer review asked for several reads 5 minutes apart and three were agreed: 15 minutes watched, and one bad reading alone never fails a fix. Lower: one bad minute decides, or a slow recovery is called failed. Higher: a later verdict and a longer Validating state. Workspace: Change safety → Post-change validation (Rounds 1–6, Seconds apart 60–1800).
Added round: 1 When round 1 read nothing (not observed) and every later round was clean, the window reads one more round in its place before the read-only verify takes over; the verdict counts the rounds after round 1 and says after 1 added round (round 1 read no data). Once per window. The first round opens on the slowest signal's period 30 s after the write, often before a started instance has published a datapoint: live on 25 October a start at 01:04:28 left round 1 (01:05–01:10) without data although the fix worked, rounds 2 and 3 were clean, and the fix then waited 83 minutes for the verify. One round is the smallest wait that still gives the configured number of rounds with data. 0: such a fix waits for the read-only verify. More: a fix whose signals stay silent reaches a person later. Fixed.
Reachability analyses: aiops_reachability_max_per_change 10 a change; 60 s each; sweep after 1 h With aiops_reachability_checks on (off by default), one analysis per neighbour with an EC2 network interface, once per window (a not observed one again); past the cap, or unfinished after 60 s, the neighbour reads not observed. While on, a sweep every 15 minutes removes aiops:change paths older than an hour. About USD 0.10 an analysis on the customer's AWS bill: a plan lists at most 9 neighbours, so 10 is one each plus a retry (about USD 1 a change). 60 s keeps a round short (AWS states no completion time: experience); an hour outlasts any analysis in use. Lower cap: more neighbours not observed. Higher: more spend. Workspace: both keys (the cap 0–60). Fixed: 60 s, the hour, 15 minutes.
Hold after a start, reboot or restart: 600 s For 10 minutes after a fix's start, reboot or restart step an automatic verify writes a $0 held row instead of starting the agent. A person's Verify still runs. EC2 status checks report initializing and services need their readiness probes for minutes after a start: an agent checking sooner reads an instance that has not finished booting and answers uncertain, at a cost. Experience. Lower: paid checks of a machine still booting. Higher: a failed restart is noticed later. Fixed.
Verify run steps: AIOPS_RECHECK_MAX_STEPS 6 The agent run that checks a fix when code could not decide. A verify reads a few named signals. Experience. Lower: more uncertain answers. Higher: more spend per check. Platform.
Check order: AIOPS_INCIDENT_RECHECK_BATCH 5 a tick; a remediating incident first The scheduled check takes 5 incidents a tick, one workspace's turn after another's, the longest unchecked first — except that an incident whose fix ran (Remediating) goes before the other incidents of its workspace. That fix is only validated by this check: by age alone it waited behind every open scanner finding of its workspace (live on 25 October: 147 open findings at 5 a tick, 83 minutes). 5 is experience. Lower: every check waits longer. Higher: more checks start at once. Platform: the batch. Fixed: the order.
Verify attempts: AIOPS_VERIFY_MAX_ATTEMPTS 10 After 10 post-fix checks that ran without a clear answer, the incident goes to a person; at the AIOPS_INCIDENT_RECHECK_INTERVAL_S 900 s cycle that is about 2.5 hours. A person's Verify now still runs after that. Stops a check that can never conclude from polling forever. Experience. Lower: slow-to-clear fixes reach a person early. Higher: an endless check costs more. Platform.
Validation plan: 10 items, at most 5 of them changed resources The change request's validation plan lists what the validation after the fix reads: each changed resource (at most 5) with its read and what it must show, the plan's own success check, then the services one hop away from the changed service, callers and callees in turn, until the plan holds 10 items. Only what the plan lists is read, and the approval covers that list. The customer review set the scope at one hop of neighbours; 10 keeps each round's reads, and the list a person approves, short. Experience. Lower: neighbours drop off the plan, and a break next door is not seen. Higher: more reads per round and a longer list to approve. Fixed.
Close without a verdict: 24 h A released change still verifying whose verdict has been none, or only uncertain, for 24 hours (from its first uncertain since the write, or from the write when no check answered) can be closed by a person with approvals.decide, with a reason. It becomes failed with the verdict uncertain; nothing rolls back on the close, and a later verdict still decides. A day leaves room for the checks after a fix and for a scanner that reports once a day. Experience. Lower: a change is closed before a slow verdict arrives. Higher: a change no check can answer stays open longer. Fixed.

Recurrence and history

Value · default Where it applies Why this value Lower / higher Who changes it
Problem record: AIOPS_PROBLEM_THRESHOLD 3 within AIOPS_PROBLEM_WINDOW_DAYS 30 days; AIOPS_PROBLEM_AUTOMATION_N 3 The 3rd same-key incident in 30 days opens or joins a problem record; a fix verified on 3 linked incidents is an automation candidate. The 30 days also bound the recurring tag and how old a verified fix may be for an automatic API fix to repeat it. Experience: one repeat can be chance; a third within a month is a pattern. Lower: one-offs become problems. Higher: recurring issues are flagged later. Platform.
Reopen window: aiops_reopen_window_h 0 (off) As in Intake rules; noise closes never reopen. Off, each occurrence is its own incident. One incident per occurrence keeps its own analysis, cost and verdict; recurrence is counted by the problem record. Higher: a new cause on the same service lands in an old incident's history. Workspace: Intake rules (Alert intake & tokens), Reopen window (hours) (0–8760).
Retention: AIOPS_RETENTION_DAYS_DEFAULT 365 days How long a closed record outlives its close; removal works in whole months. Its per-record costs go with it; the monthly billing totals stay. One year keeps a full yearly review and a year of per-record costs visible. Experience; a workspace with a legal retention period sets its own. Lower: older evidence is gone when asked for. Higher: more storage, records kept longer than needed. Workspace: Data retention, retention_days (30–3650).

Scanner pull and spend

Value · default Where it applies Why this value Lower / higher Who changes it
Scanner pull: aiops_scanner_sync_interval_s 900 s Scheduled pull of Security Hub and Defender for Cloud findings through the read connections; Amazon Inspector findings too, unless the AWS connection's Pull Inspector vulnerability findings (CVEs) (Advanced connection settings) is false. Pull and webhook share deduplication. On, because the cloud's own scanners are the detection layer: a finding reaches the platform within one cycle. Each HIGH or CRITICAL finding, CVEs included, arrives as P1/P2 and, while Analyse P1/P2 findings automatically is on (the default), starts one paid analysis. Shorter interval: more Security Hub calls (throttled calls are retried). Longer: findings arrive later. Workspace: Scanner sync, scanner_sync_interval_s (0 = off, or 300–86400). The Inspector field: the AWS connection.
Per-run cap: max_cost_usd, platform AIOPS_MAX_COST_USD_DEFAULT 0 (uncapped) Checked before each step. At 70 % of the cap the remaining steps move one model tier down (when tiers are set); at the cap the run stops without a summary and the incident escalates. Measured in the test lab (August 2026): a run on the heaviest model used cost USD 0.39 at the median and USD 1.36 at the 90th percentile (all run types; upper bounds, as cached prompt reads were billed at the full input price). A cap several times the 90th percentile (for example USD 10) stops a looping run, not a hard incident. Lower: hard incidents stop before their summary. Higher: a looping run costs more. Workspace: Cost cap & PII patterns, max_cost_usd. Platform: the default.
Daily budget: daily_cost_limit_usd, platform AIOPS_DAILY_COST_LIMIT_USD_DEFAULT 0 (off) Checked when a run starts; once the day's spend (UTC) reaches it, no new analysis starts and incidents wait for a person. A run already started finishes, so a day can end above the limit by up to one run cap per run in flight. Off by default: the right limit depends on the workspace's incident volume. Size it from the expected incidents a day times the cost per analysis. Lower: analyses stop earlier on a busy day. Higher: a burst costs more before anyone looks. Workspace: AI budgets & kill switch, daily_cost_limit_usd.

Findings, patching and backup

Value · default Where it applies Why this value Lower / higher Who changes it
Raise from context: one tier up (at most L3) on an internet-facing or sensitive-data resource; CVSS 9 or more on an internet-facing resource = critical A scanner finding on an internet-facing resource, or on one that holds sensitive data, has its fix tier raised one step; a CVE with a CVSS score of 9 or more on an internet-facing resource is also raised to critical, which opens at P1. A fix the raise lands at L3 waits for a person. Raise only: nothing is lowered. 9.0 is where CVSS v3 starts its Critical band; on a resource reachable from the internet such a CVE can be exploited without a foothold inside. Lower (for example 7): most HIGH CVEs on public resources become P1. Higher: none is raised. Fixed. Exposure and sensitivity come from the resource's tags, the finding's details and the workspace's risk_context.
Patch compliance card: target 100 %; patch age aiops_patch_within_days 0 (no rule) The share of AWS Systems Manager managed nodes Patch Manager reports compliant; EC2 instances Systems Manager does not manage are listed apart. With a patch-age rule, a node whose last patch operation ended longer ago counts as non-compliant too; without one the card shows 40 days as an example, with the nodes it would flag. Each non-compliant node also opens a Patching finding, unless Security Hub's SSM.2 already holds it. Read daily (AIOPS_COMPLIANCE_REPORT_INTERVAL_S 86400 s) and on Refresh reports. A lower target would accept a known missing patch as normal. No patch age by default on purpose: the number is the customer's to name. — Fixed: the target and the 40-day example. Workspace: aiops_patch_within_days (0–3650), API only.
Backup card: AWS Backup jobs over the last 24 hours and the last 7 days; target 100 % The share of finished AWS Backup jobs that completed, in each window, next to coverage (every EC2 instance in a backup plan, every RDS database with backup retention). Read daily, like the card above. 24 hours shows last night's run; 7 days shows whether failures repeat, and includes a weekly plan. Our choice. Shorter: one bad night decides the figure. Longer: a new failure hides behind older successes. Fixed.
Pre-change backup: aiops_backup_before_change on; kept 7 days (aiops_backup_retention_days); waited for up to 30 minutes Before a reboot-type step on an EC2 instance or an Azure VM, an AMI (taken without a reboot) or an OS-disk snapshot, waited for until it is available and tagged aiops:retain-until with the date it may go; no usable backup, no change. The platform never deletes a backup: your own clean-up removes it after that date. 7 days covers the validation rounds, the automatic rollback check in the 24 hours after a failed fix, and a weekend before anyone looks. Experience. Shorter: a fix that breaks later has no backup to restore from. Longer: the backups cost more until someone removes them. Workspace: Change safety, Back up a stateful target before a reboot-type change and Keep pre-change backups for (days) (1–365). Fixed: the 30 minutes.
Recommended backup plan: daily at 18:00 UTC (02:00 in Singapore), recovery points kept 35 days The plan a backup finding's commands create for an EC2 instance or RDS database with no recovery point. 18:00 UTC is outside business hours in Singapore; 35 days follows AWS Backup's own plan template Daily-35day-Retention. Our proposal until your backup policy names its own. Shorter retention: a problem found late may have no clean recovery point. Longer: higher backup storage cost. Workspace: aiops_backup_plan_schedule (an AWS Backup cron(...) or rate(...) expression) and aiops_backup_plan_retention_days (0 = keep until deleted), API only.
Image reminder: aiops_image_reminder_h 24 h While an image the platform rebuilt under a new tag is not deployed (the scanner still reports the old digest, and the incident is open), targets that take image_not_deployed are reminded this often after the first message, which the validation sends. A rebuild more than 90 days old is no longer looked at. Once a day reaches whoever deploys without a message on every scheduler tick. Experience. Lower: a run of reminders on a slow deploy. Higher: a forgotten deploy keeps the old, vulnerable digest in use longer. Workspace: aiops_image_reminder_h (0–720, 0 = the first message only), API only. Fixed: the 90 days.

Intake and change-record limits

Value · default Where it applies Why this value Lower / higher Who changes it
Webhook body: AIOPS_WEBHOOK_MAX_BODY_BYTES 1 MiB; 8 MiB for Alertmanager, Grafana and the generic intake; 16 MiB for the Kubernetes audit webhook A larger body is refused with 413 before it is read; each refusal is logged with the source and the size. Before 0.1.149 any size the load balancer let through was read into memory before any check. 1 MiB fits the sources that send one alert or finding at a time. Alertmanager, Grafana and the generic intake send a whole alert group in one request, and a storm of a few thousand alerts passes 1 MiB; the Kubernetes API server batches its audit events. Our choice. Lower: a large alert group is refused, and a sender does not retry a 413 (Alertmanager's max_alerts keeps a notification under the cap). Higher: one large body takes more memory on the control plane. Platform (0 = no cap). Fixed: the 8 and 16 MiB, which a larger platform value still raises.
CI/CD writes in Kubernetes: 50 objects per cluster and service account per clock hour Writes by service accounts in the workspace's CI/CD namespaces are recorded as changes made by a service, one row per object; past 50 in the hour the rest are counted, not recorded one by one. Denied attempts and people are never capped. A release of one application touches a handful of objects (its deployment, config maps, services), so it is recorded in full, while a bulk run, such as a sync of a whole cluster, cannot fill the change list. Experience. Lower: a large release is cut short in the change record. Higher: one pipeline run fills the change list. Workspace: the namespaces, k8s_ci_namespaces (default flux-system, jenkins, gitlab-runner, tekton; at most 20), API only. Fixed: the 50 and the hour.
Denied access: one SecOps incident per principal, account and hour A cloud API call the provider refused is unauthorized access; nothing changed. While the SecOps agent is enabled, one P3 SecOps incident opens per principal and account in each hour, one alert per attempt inside it, with no analysis. A script that retries a denied call would otherwise open an incident per attempt; one hour holds one burst, and a new incident in the next hour shows the attempts go on. Experience. Shorter: more incidents for one burst. Longer: a new attempt hours later hides in an old incident. Workspace: Unmanaged changes, Report denied access attempts to SecOps (aiops_denied_access_to_secops, on). Fixed: the hour.

Built-in detection

Value · default Where it applies Why this value Lower / higher Who changes it
Poll: AIOPS_BUILTIN_DETECTION_POLL_S 300 s One evaluation per workspace per poll. Rules use 1-minute data (EC2: 5-minute); 5 minutes keeps the read cost low and still sees a breach within one poll. Lower: higher CloudWatch read cost. Higher: later alerts. Workspace: Built-in detection offers 60, 300 or 900 s; poll_s takes 60–3600 through the API.
Data delay: AIOPS_BUILTIN_DETECTION_DELAY_S 600 s The newest datapoint read is 10 minutes old. CloudWatch data lands late; Datadog's CloudWatch monitors use 900 s. Lower: late data reads as missing and alerts flap. Higher: later alerts. Platform.
Clear: 3 clean polls in a row An alert ends after 3 polls with no breach; a failed read is not clean. Azure's metric-alert resolve rule; stops close-and-reopen on one good datapoint. Lower: flapping. Higher: incidents stay open longer after recovery. Fixed.
Shadow: AIOPS_BUILTIN_DETECTION_SHADOW_H 24 h After turning it on, rules are evaluated but send nothing. One day, including the daily peak, shows which rules fire on normal behaviour. Experience. Lower: noisy rules alert before they can be turned off. Higher: real problems wait longer. Workspace: shadow_hours (0–168), through the API only.
Resource cap: AIOPS_BUILTIN_DETECTION_MAX_RESOURCES 300 Resources per poll, round-robin across kinds. Bounds the CloudWatch read bill; the card shows the estimate. Lower: resources unwatched. Higher: higher cost and longer evaluations. Workspace: Built-in detection, max_resources (1–5000).
Baseline (R26–R28): 95th percentile of 5-minute values over 7 days, read once a day, at least 288 values What a latency or fault-rate rule compares against, per series. A week holds the weekly cycle; the 95th percentile ignores one bad hour of it; 288 five-minute values are one day, so the daily peak is in it; daily, because a baseline moves slowly and 50 series × 7 days of 5-minute values fill one GetMetricData call (R26 reads two series per service, so two calls). AWS's TargetResponseTime alarm says to set the threshold from "the historical behavior of this metric". Ours. Shorter window or lower percentile: normal peaks alert. Longer or higher: a slow regression hides in it. Fixed.
Baseline margin: the larger of baseline × 1.5 and baseline + 100 ms (R27, R28); of 5 % and baseline + 5 points (R26) The threshold above the baseline. Two floors so neither a fast service's jitter (20 ms to 40 ms is no outage) nor a slow service's normal swing alerts; 100 ms is about the smallest delay a user notices. 5 % as R11 and R12. Ours. Lower: alerts on normal variation. Higher: a real slowdown alerts later. Fixed.
Baseline rule window: 3 of 3 × 5 min R26–R28 and R30. The baseline is made of 5-minute values, so the rule reads the same resolution; 15 minutes in a row is a degradation, not one slow garbage collection. A spike that recovers after 15 minutes still raises. Ours. Shorter: one slow period alerts. Longer: later alerts. Fixed.
Series per resource: 100 a poll The pods and services read for one EKS cluster (R26, R27, R29, R30), shared round-robin across the rules, the rule with the fewest series first. Bounds one cluster's read cost: 100 series every 5 minutes is about US$ 8.60 a month; round-robin, so a cluster's services never crowd out its one failed-node series. Ours. Lower: big clusters' tails unwatched. Higher: cost grows with the cluster. Fixed.
New rules in shadow: the workspace's shadow period (shadow_hours, 24 h) from the first poll the rule covers something A rule added by a release after the workspace armed. An upgrade never starts a rule firing unseen: one day of its would-have-fired hits shows whether it is noise before it opens anything. — Workspace: Arm now arms them at the next poll; shadow_hours.
Kubernetes waits: Pending 10 min (R33), not Ready 5 min (R34) How long a pod may stay Pending, or a node not Ready, before it is an alert. R33: cluster autoscaling adds a node and a large image pulls within a few minutes; 10 minutes is past both. Ours. R34: Kubernetes' own default — a not-ready node's pods are evicted after 300 s (the node.kubernetes.io/not-ready toleration). Lower: a normal scale-up or a slow pull alerts. Higher: a stuck pod or a dead node alerts later. Fixed.
Kubernetes read: 15 s a call, 90 s a cluster, 40 kubectl calls plus one per namespace listed on the connection One cluster's read per poll (pods, nodes, namespaces), beside the CloudWatch reads; every cluster at the same time. A cluster that does not answer gives up after its first call (15 s), and clusters are read side by side, so slow API servers never add up past the evaluation's 240 s timeout; the calls leave room for the namespace-by-namespace fallback of a namespace-scoped account, and a read the cap cuts short keeps every state. Experience. Lower: big or namespace-scoped clusters fail their read. Higher: a dead cluster holds a worker thread longer. Fixed.
State alerts open: 25 per connection (per account and region for scheduled maintenance, per account for provider health) R31–R37. A broken cluster or a host-reboot campaign is one line on the card, not hundreds of incidents; the rest open as those clear. Ours. Lower: real failures wait behind open ones. Higher: one bad cluster floods the incident list. Fixed.
Scheduled maintenance: read hourly, one clean read ends it, severity info (P4 unless criticality or blast radius raise it) EC2 instance events and RDS pending maintenance (R35, R36). AWS announces them days or weeks ahead, so an hour early is early enough and costs one call per account, region and service an hour; the event is a state AWS ends itself (completed, canceled), so one read without it is the end; P4 because nothing has failed yet — it is planned work on the resource. Ours, a proposal: whether it should be an incident or a notice only is still open. Shorter: more calls, nothing earlier that matters. Longer: an event announced at short notice is seen later. Fixed.
Provider health: every poll, cached 5 min, severity error (P2) AWS's open issues for the services and regions of the in-scope estate (R37). A provider outage is time-critical: read with every poll, but at most once per account every 5 minutes (the public dashboard once per set of regions and services; a role without health:DescribeEvents reads the public dashboard and is asked again an hour later), so a 1-minute poll adds no calls. P2 because the provider is degraded where the estate runs, not yet proven to hurt it; an incident only where the estate is exposed. Ours. Shorter cache: more Health calls, nothing sooner that matters. Longer: an issue is seen later. Fixed.
Rules R01–R37 Thresholds and windows in Built-in detection. AWS Recommended alarms (Classic), the ALB and NLB metric pages and the OpenSearch recommended alarms. Severity is the platform's choice. Ours (no vendor source gives the number): R09 20 ms; R10 and R13 the 1-minute period (R13 also 2 of 2); R11 and R12 5 %; every rate rule's floor of 10 requests or invocations a minute (R11, R12, R14, R16); R18 50 % of retention; R19 > 0. R14's 5 % follows Dynatrace's recommended AWS rule, where AWS alarms on errors above 0. R26–R30 (0.1.153): R26 5 %; R26, R27 and R28's floor of 10 requests a minute; R28 the p95 statistic; R29 ≥ 1 failed node (any node condition, in AWS's 5 of 5 × 1 min window for EKS Container Insights alarms); R30 ≥ 1 restart in each 5-minute period, because kubelet's crash-loop back-off caps at 5 minutes; R31's 3 restarts between back-offs; R33's 10 minutes. R31–R34 alert on the first read that sees the state (CrashLoopBackOff follows a failed restart). Every new rule is a warning. Lower thresholds or shorter windows: alerts on normal behaviour. Higher or longer: real failures alert later or not at all. Workspace: turn a rule off (rules_off), and Warning rules start an analysis (warning_investigate). Thresholds are fixed.

Synthetic checks

Value · default Where it applies Why this value Lower / higher Who changes it
Interval: 300 s (60–600) How often a check probes its target from the control plane. The customer review asked for every 1 to 5 minutes; 5 minutes is the upper end, so a suggestion probes a customer endpoint at most 12 times an hour. Lower: an outage is seen sooner, more requests to the endpoint. Higher (up to 600): an outage is seen later. Workspace: Every (seconds) on each check.
Timeout: 10 s (1–30) No answer within it is a failed run. Experience: a page slower than 10 s is down for its users. Lower: a slow endpoint reads as down. Higher: a hung endpoint holds the run longer. Workspace: each check (API timeout_s).
Retry: one, 10 s later A failed run is tried once more; only a failure that repeats counts. A spike that recovers within 10 s raises nothing. Our proposal: one dropped connection or a single 502 during a deploy is not an outage. The customer asked that a spike which recovers still raises; that is open with them. Lower (no retry): every blip opens an incident. Higher: a real outage is confirmed later. Fixed.
Back to healthy: 2 passing runs in a row A check that failed turns healthy, and its alert resolves, after two passes. One pass after a failure can be the same flapping; two is the smallest run that is not. Experience. Lower: close and reopen on a flapping endpoint. Higher: incidents stay open longer after recovery. Fixed.
Severity: critical for your own checks, warning for a suggestion you keep The severity of the alert a confirmed failure sends. A check you added watches something you named as important; a kept suggestion was picked by the platform. — Workspace: Severity on each check.
Suggestions: aiops_monitor_derive on; at most 20; expected status 200-499 Once a day the platform suggests a check for each internet-facing load balancer and API Gateway API in the workspace scope. A suggestion probes its endpoint (GET / every 300 s) in trial and opens nothing until a person keeps it. The customer review asked for checks by default; any answer below 500 means the endpoint answered (403 or 404 at / is normal). 20 bounds the requests a workspace sends without anyone asking. Off: no new suggestion; one already listed keeps probing until it is discarded, unless its resource falls outside the scope (removed at the daily run). Workspace: aiops_monitor_derive. Fixed: 20 and 200-499.
Checks per workspace: 100 Every check, suggestions included. Bounds the probes one workspace sends from the control plane. Experience. — Fixed.

Next steps