# Built-in detection (AWS)

> CloudWatch rules over your estate inventory for accounts with no alarms — outages, and latency or faults above each service's own baseline; read-only, shadow first.

_Concept · Updated 2026-10-28 · Describes release 0.1.164_

Built-in detection watches the AWS resources in your estate inventory with a fixed set of
CloudWatch rules, for accounts where nobody created alarms. It is **off** until a workspace
owner turns it on, starts in **shadow mode**, and only reads: it lists metrics and reads their
datapoints, and never creates an alarm, an anomaly detector or anything else in your account.
No model decides anything here. A rule that fires sends an ordinary alert (source `builtin`)
into the same pipeline as your own alerts: deduplication, correlation, grouping, the
incident and, for a critical rule, an investigation.

Where it lives: **Settings · Data in → [Datasources](/portal/datasources) (in the product: `/portal/datasources`) → Built-in detection (AWS)**.

## What it watches

Thresholds and M-of-N come from AWS's *Recommended alarms (Classic)* list, the ALB and NLB
metric pages and OpenSearch's recommended alarms. *Ours* marks a number no vendor source gives;
every number and its reason is also in [Thresholds and why](/help/thresholds-and-why.md#thresholds-builtin-detection).
R26 to R30 (0.1.153) watch degradation as well as outages, where nobody created an alarm: a
service answering slower or failing more than it normally does is an incident too.
"Gate *self*" means the rule runs only where its own metric is published (a conditional metric);
"gate *resource*" means it runs on every resource of its kind that published anything in the
last 3 hours (its counts are published only when non-zero).

| ID | Resource | Condition | Window | Severity | Needs from the inventory |
|---|---|---|---|---|---|
| R01 | EC2 instance | `StatusCheckFailed` Max ≥ 1 | 2 of 2 × 5 min | critical | — |
| R02 | EC2 instance | `StatusCheckFailed_AttachedEBS` Max ≥ 1 | 10 of 10 × 1 min | critical | — (Nitro only, by gate) |
| R03 | EBS volume, through its instance | `VolumeStalledIOCheck` Max ≥ 1 | 10 of 10 × 1 min | critical | — |
| R04 | EC2 instance | `CPUUtilization` Avg > 80 % | 3 of 3 × 5 min | warning | — |
| R05 | RDS instance | `CPUUtilization` Avg > 90 % | 5 of 5 × 1 min | warning | — |
| R06 | RDS instance (not Aurora) | `FreeStorageSpace` Min < 10 % of allocated storage | 5 of 5 × 1 min | critical | allocated storage; skipped with storage autoscaling |
| R07 | RDS / Aurora instance | `FreeableMemory` Avg < 25 % of class memory (Aurora 5 %) | 15 of 15 × 1 min | warning | instance class (`db.m*`, `db.r*`, `db.t*`) |
| R08 | RDS read replica | `ReplicaLag` Max > 60 s | 10 of 10 × 1 min | warning | replica source (never on a primary) |
| R09 | RDS instance | `ReadLatency` or `WriteLatency` p90 > 20 ms (*ours*) | 5 of 5 × 1 min | warning | — |
| R10 | ALB target group | `UnHealthyHostCount` Min > 0 | 2 of 2 × 1 min (*ours*: the 1-minute period) | critical | — |
| R11 | ALB | `HTTPCode_ELB_5XX_Count` ÷ (`RequestCount` + `HTTPCode_ELB_503_Count`) > 5 %, ≥ 10 a minute (*ours*) | 5 of 5 × 1 min | critical | — |
| R12 | ALB target group | `HTTPCode_Target_5XX_Count` ÷ `RequestCount` > 5 %, ≥ 10 a minute (*ours*) | 5 of 5 × 1 min | warning | — |
| R13 | NLB target group | `UnHealthyHostCount` Min > 0 or `HealthyHostCount` Max < 1 | 2 of 2 × 1 min (*ours*) | critical | — |
| R14 | Lambda function | `Errors` ÷ `Invocations` > 5 %, ≥ 10 a minute (*ours*: the floor) | 3 of 3 × 1 min | warning | — |
| R15 | Lambda function | `Throttles` Sum ≥ 1 | 5 of 5 × 1 min | warning | — |
| R16 | API Gateway REST or HTTP API stage | 5xx count ÷ `Count` > 5 %, ≥ 10 a minute (*ours*: the floor) | 3 of 3 × 1 min | critical | the REST API's name |
| R17 | API Gateway REST or HTTP API stage | `Latency` p90 ≥ 2500 ms | 5 of 5 × 1 min | warning | the REST API's name |
| R18 | SQS queue | `ApproximateAgeOfOldestMessage` Max ≥ 50 % of retention (*ours*) | 15 of 15 × 1 min | critical | retention period |
| R19 | DynamoDB table | `ReadThrottleEvents` or `WriteThrottleEvents` Sum > 0 (*ours*) | 5 of 5 × 1 min | warning | — |
| R20 | EFS file system | `PercentIOLimit` ≥ 100 % (General Purpose) or `BurstCreditBalance` ≤ 0 (Bursting) | 15 of 15 × 1 min | warning | performance and throughput mode |
| R21 | OpenSearch domain | `ClusterStatus.red` Max ≥ 1 | 1 of 1 × 1 min | critical | — |
| R22 | OpenSearch domain | `ClusterIndexWritesBlocked` Max ≥ 1 | 1 of 1 × 5 min | critical | — |
| R23 | OpenSearch domain | `FreeStorageSpace` Min ≤ 25 % of node storage | 1 of 1 × 1 min | warning | EBS volume size per node |
| R24 | OpenSearch domain | `JVMMemoryPressure` Max ≥ 95 % or `OldGenJVMMemoryPressure` Max ≥ 80 % | 3 of 3 × 1 min | warning | — |
| R25 | NAT gateway | `ErrorPortAllocation` Sum > 0 | 15 of 15 × 1 min | warning | — |
| R26 | Application Signals service on an EKS cluster | `Fault` ÷ `Latency` SampleCount > the larger of 5 % (*ours*) and its baseline + 5 points (*ours*), ≥ 10 requests a minute (*ours*) | 3 of 3 × 5 min (*ours*) | warning | the EKS cluster |
| R27 | Application Signals service on an EKS cluster | `Latency` p90 > the larger of its baseline × 1.5 and baseline + 100 ms (*ours*), ≥ 10 requests a minute (`Latency` SampleCount, *ours*) | 3 of 3 × 5 min (*ours*) | warning | the EKS cluster |
| R28 | ALB | `TargetResponseTime` p95 (*ours*) > the larger of its baseline × 1.5 and baseline + 100 ms (*ours*), ≥ 10 requests a minute (`RequestCount`, *ours*) | 3 of 3 × 5 min (*ours*) | warning | — |
| R29 | EKS cluster (Container Insights) | `cluster_failed_node_count` Max ≥ 1 (*ours*) | 5 of 5 × 1 min | warning | the EKS cluster |
| R30 | Pod of an EKS cluster (Container Insights) | `pod_number_of_container_restarts` Sum ≥ 1 in each period (*ours*) | 3 of 3 × 5 min (*ours*) | warning | the EKS cluster |
| R31 | Kubernetes workload (kubectl) | a container of one of its pods waits in `CrashLoopBackOff`, or restarted 3 times or more (*ours*) and crashed again since the previous read | each poll | warning | a kubectl connection |
| R32 | Kubernetes workload (kubectl) | a container of one of its pods terminated `OOMKilled` since the previous read | each poll | warning | a kubectl connection |
| R33 | Kubernetes workload (kubectl) | a pod in phase `Pending` for more than 10 minutes (*ours*) | each poll | warning | a kubectl connection |
| R34 | Kubernetes node (kubectl) | `Ready` condition not `True` for more than 5 minutes | each poll | warning | a kubectl connection |
| R35 | EC2 instance | an `instance-retirement`, `instance-stop`, `instance-reboot`, `system-reboot` or `system-maintenance` event AWS scheduled, not completed or canceled | hourly | info (P4, *ours*) | — |
| R36 | RDS DB instance | a pending maintenance action with a date (the next maintenance window, applied automatically after a date, or forced on a date) | hourly | info (P4, *ours*) | — |
| R37 | AWS service in a region | an open AWS Health issue for a service and region where the estate has an in-scope resource of that service | each poll | error (P2) | any resource with an ARN |

Each row on the card links to its source page. A rule AWS says not to apply is skipped and the
card says why (for example "storage autoscaling on", "not a read replica", "retention not read").

### Baselines (R26, R27, R28) {#baselines}

Latency and fault rates have no number that fits every service: 300 ms is normal for one API and
an outage for another. These three rules compare each series with **its own baseline**: the
95th percentile of its 5-minute values over the last 7 days. A week holds the weekly cycle, and
the 95th percentile lets one bad hour of it pass without hiding a regression. The threshold is
the larger of two floors, so neither a fast service's jitter nor a slow service's normal swing
alerts: for latency, 1.5 × the baseline and the baseline + 100 ms; for the fault rate, 5 % and the
baseline + 5 points. AWS's own TargetResponseTime alarm says the threshold "depends on your
situation" and to "analyze the historical behavior of this metric": the baseline is that history.
A 5-minute period under 10 requests a minute is no evidence, for the baseline or the rule: a
low-traffic service's p90 is one slow request.

- A series needs **a full day of data** (288 five-minute values) before it has a baseline; until
  then the rule is not applied to it and the rule table says *baseline still building*.
- The baseline is **read once a day** per series (one GetMetricData call carries a week of 50
  series; R26 reads two per service, its faults and its request count), and
  never while the series is in alarm: an ongoing degradation does not become its own normal.
  A degradation that lasts long enough to fill 5 % of a week (about 8 hours) and then clears
  raises the baseline for the following week.
- A breach must last 3 consecutive 5-minute periods: a spike that recovers within 15 minutes
  raises nothing. One that lasts 15 minutes is an episode whether or not it recovers afterwards,
  and its incident stays on record.

An alert on an inventory resource (an instance, a load balancer, a database …) carries the
resource's own service as its `service`: the first of its `service`, `app`, `application` or `Name`
tags (keys in any case), else the resource's name. An instance tagged `service=web` therefore joins
web's other alerts on one incident; the alert's summary still names the resource.

### Application Signals and Container Insights on EKS {#eks}

Application Signals services and Container Insights pods are not inventory resources. Their
series belong to the **EKS cluster** they run in: an Application Signals environment
`eks:<cluster>/<namespace>` names it, Container Insights' `ClusterName` dimension too. The cluster
must be in the estate inventory (a tagged cluster is; Resource Explorer lists untagged ones) and
inside the scope: its tags decide for every service and pod on it. Services of other environments
(EC2, ECS, Lambda) are counted as series with no inventory resource. The alert's `service` is the
series' own `Service` (R26, R27) or `PodName` (R30, the workload's name) dimension, and the
cluster's name for R29, so a fault, a latency rise and a restarting pod of one service land on one
incident. One resource carries at most **100 series** a poll, shared round-robin across the rules,
the rule with the fewest series first, so a cluster's many services never crowd out its one
failed-node series (R29); the series past that are counted as *over the 100 series per resource
cap*.

These rules follow the **cluster's** tags: on a cluster in scope, every Application Signals service
and every pod is watched, whichever team's namespace it runs in. To keep to your own namespaces on a
shared cluster, turn R26, R27 and R30 off and use the Kubernetes rules below, which read the
namespace's labels.

R30 counts restarts in each 5-minute period (the CloudWatch agent reports the restarts since its
previous collection): kubelet's crash-loop back-off caps at 5 minutes, so a container that keeps
crashing restarts at least once in every period, and 3 periods in a row is 15 minutes of it.

### Kubernetes state (R31 to R34) {#kubernetes}

A cluster's own state says what its metrics may not: a pod crash-looping, killed out of memory,
stuck Pending, a node gone NotReady. Every poll, built-in detection reads the cluster through the
workspace's **read-only kubectl connection** (`kubectl get pods`, `get nodes`, and `get namespaces`
when a scope needs their labels — read verbs only, the same reader as the estate inventory, its
namespace-by-namespace fallback included). This works for any cluster the connection reaches,
EKS or not, and needs no Container Insights.

- **One alert per workload**, not per pod (pods change names): the workload is the pod's
  controller — a Deployment (through its ReplicaSet), StatefulSet, DaemonSet or Job; a bare pod is
  its own. Its `service` label comes from the workload's labels (`app.kubernetes.io/name`, then
  `app`, then `k8s-app`), else the workload's name. The alert also carries `namespace`, the
  workload (`deployment=payments`), one `pod`, the `container`, the `cluster` and `resource_id`
  `k8s:<cluster>/<namespace>/<Kind>/<name>`, the estate inventory's id for it. `<cluster>` is the
  connection's name (or a `cluster` value in its configuration), so two connections never share an
  id; the next inventory harvest removes the nodes an earlier version wrote as `k8s:cluster/…`, and
  an alarm under such an id keeps clearing the usual way. One alert per node for R34.
- **R31** also counts a container between back-offs: after each crash the kubelet shows it
  terminated for a minute or so, then running again, and at the 5-minute back-off a poll can land
  there every time. A container that restarted 3 times or more and exited with an error since the
  previous read is a crash loop too; one lost with its node (reason `Unknown` after a reboot) is not.
- **R32** counts an out-of-memory kill once, in the read after it happened. **R33** waits
  10 minutes (cluster autoscaling and a large image pull usually finish sooner) and says why the
  pod waits (`Unschedulable: 0/3 nodes are available…`, `ImagePullBackOff`). **R34** waits
  5 minutes, the time after which Kubernetes itself starts evicting a not-ready node's pods.
- **Scope:** a Kubernetes label counts as a tag. With a workspace scope, a pod is watched when it
  or its namespace carries every scope tag as a label (for example `aiops-managed: "true"` on the
  namespace); a node when it runs a watched pod, and while its own alert is open (Kubernetes
  evicts a not-ready node's pods after 5 minutes, the same moment R34 fires). Without a scope,
  everything the connection can read is watched.
- **A failed read** keeps every state as it is; the card's **Other reads** line says why. A read
  fails when the pods cannot be listed (forbidden, or no answer: each kubectl call gives up after
  15 seconds, a cluster after 90, and a cluster whose pods do not answer is not asked for anything
  else), or when the call cap cut it short (40 calls, plus one per namespace listed on the
  connection). Clusters are read at the same time, not one after another. When only the
  namespaces cannot be listed, the scope is read from the pods' own labels and a workload already
  in alarm stays watched. A workload with no pod left, in a namespace the read listed and still
  inside the scope, has cleared: it resolves at once like a recovered one, so its incident can
  close on its own after the quiet period. A node that no longer exists, or a workload or node that
  left the scope, resolves at once with a note (nothing was seen to clear); a recovered one after
  three clean reads.
- **At most 25 alerts open per connection:** a broken cluster opens 25, and the card's **Other
  reads** line says how many more are not alerted; the rest open as those clear. The same cap
  holds for each account and region's scheduled maintenance and each account's provider issues.
- Every enabled read kubectl connection is read while built-in detection is on. To leave one out,
  set `builtin_detection` to `false` in its configuration (API: `PATCH
  /api/aiops/{tenant_id}/datasources/{ds_id}` with the whole current config and that key — a
  config patch replaces the config); the connection dialog has no box for it yet.

### Scheduled maintenance (R35, R36) {#scheduled-maintenance}

AWS announces maintenance on a resource days or weeks ahead: an instance on degraded hardware is
retired, a host is rebooted, a database gets an operating-system update. Nothing has failed yet,
so these open a **P4** incident on the resource itself (severity `info`: held, no paid analysis),
in time to plan the change instead of meeting it. A mission-critical resource or a blast radius of
2 or more services raises the priority, as for any incident. P4 is our proposal: whether scheduled
maintenance should be an incident at all, or a notice only, is still an open question. Once an hour, for each account and region with an
in-scope EC2 instance or RDS database in the inventory, the evaluating AWS connection reads:

- **`ec2:DescribeInstanceStatus`** for the instances with a scheduled event (any instance state);
  one alert per instance and event code, with the window AWS gives (`not before`, `not after`) and
  the deadline. A description AWS marks `[Completed]` or `[Canceled]` is history and clears it.
- **`rds:DescribePendingMaintenanceActions`**; an action with a date — applied in the next
  maintenance window, automatically after a date, or forced on a date — is one alert per database
  and action. An action AWS files on an Aurora cluster is an alert on each of its instances in the
  inventory. An action with no date is offered, not scheduled, and raises nothing.

These are the EC2 and RDS APIs rather than the AWS Health API: they carry the same scheduled events
for every account (the Health API needs a Business, Enterprise On-Ramp or Enterprise support plan) and name the
resource directly. Only resources in the estate inventory and inside the scope are matched. One
clean read ends an episode (the event completed or was canceled). A denied read keeps its rule's
states and the card's **Other reads** line names the permission.

### Provider health (R37) {#provider-health}

When AWS itself has a problem, your alarms fire on the symptoms; R37 names the cause. Every poll,
for each AWS account with an evaluating connection, built-in detection reads AWS's open issues for
the services and regions the account's in-scope estate uses: the **AWS Health API**
(`health:DescribeEvents`, on the global endpoint) through the connection, or, on an account without
a Business, Enterprise On-Ramp or Enterprise support plan or a role without that permission, the
**public AWS Health
Dashboard**, which is not account-specific (the card's **Other reads** line then says *needs
health:DescribeEvents*, and the Health API is asked again an hour later). An issue becomes an alert only where the estate is exposed: an in-scope resource
of that service in that region (any service for an issue AWS files under *multiple services*).

- One **P2** incident per issue (severity `error`): it is analysed like a critical alert, files
  its ticket and notifies. The alert's `service` is `aws-<service>-<region>`
  (`aws-ec2-ap-southeast-1`), its message names the exposed resources (up to 5, and how many), and
  it carries `aws_service`, `health_event` and which source answered. The automated triage's
  cloud-health step reads the same issue for any other incident it touches.
- The read is cached for 5 minutes per account (the public dashboard per set of regions and
  services), so a 1-minute poll does not read it more often. An issue AWS closes clears after three
  clean polls. An unreachable read keeps the states and the card says why.
- The read goes out from the platform itself, never from a private-network agent. On a network
  without internet, the Health API (`health.us-east-1.amazonaws.com`) needs an interface endpoint
  for AWS Health (`com.amazonaws.us-east-1.health`, offered cross-Region to Asia Pacific
  (Singapore); confirm it in your account with `aws ec2 describe-vpc-endpoint-services
  --service-regions us-east-1 --service-names com.amazonaws.us-east-1.health`) plus one for STS;
  the public dashboard has no VPC endpoint and needs internet egress. A read that gets no answer
  says which: `no answer from the AWS Health API` or `no answer from the public AWS Health
  Dashboard`, then what it needs. The automated triage's cloud-health step says the same.
- Which services count: those AWS Health reports on and the inventory records with an ARN — EC2,
  EBS, RDS, Elastic Load Balancing, Lambda, SQS, SNS, DynamoDB, S3, ECS, EKS, ElastiCache, API
  Gateway and Kinesis.

### New rules start in shadow {#new-rules-start-in-shadow}

A workspace that was already armed when a release added rules (R26 to R37 in 0.1.153) does not
start them firing unseen. Each new rule watches **in shadow** from the first poll it covers
something, for the workspace's shadow period (24 hours by default): what it would have fired is
recorded under *Would have fired (shadow)*, the rule table says *New rule: shadow until* and when,
and the card names the rules. When its period ends the rule arms itself; what it already had in
alarm then opens with no analysis ("in alarm before built-in detection armed"), as at a
workspace's own arming. **Arm now** arms them at the next poll with the usual severity rules. A
workspace that turns detection on after the upgrade arms every rule together.

## What it reads, and what it never does

- **The estate inventory.** Resources come from the inventory harvest (Estate x-ray): every
  tagged resource, and since this release every EFS file system, OpenSearch domain, DynamoDB
  table, SQS queue, NAT gateway and API Gateway API, tagged or not, with the attributes the
  rules above need. A resource the inventory does not hold is not watched; the card counts the
  CloudWatch series it found with no inventory resource behind them. The list is re-read only
  when a harvest lands (daily by default, or **Harvest now** on the connection).
- **`ListMetrics`** with `RecentlyActive=PT3H`, one call per metric and namespace, cached for
  an hour: a resource is alive when CloudWatch lists it, whatever the inventory's flags say.
- **`GetMetricData`**, in batches of up to 500 metrics, every poll (5 minutes by default; the
  card offers 1, 5 or 15 minutes and the API accepts up to an hour — every 1-minute datapoint is
  evaluated whatever the poll).
- **Never** a write: no alarm, no anomaly detector, no metric stream, no `ANOMALY_DETECTION_BAND`.

## Permissions

| Read | Why | Where it is granted |
|---|---|---|
| `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData` | the rules | `ReadOnlyAccess` and `CloudWatchReadOnlyAccess`; the CloudShell read role (its own region); the platform's inline read policy names them |
| `elasticfilesystem:DescribeFileSystems`, `es:ListDomainNames`, `es:DescribeDomains`, `dynamodb:ListTables`, `sqs:ListQueues`, `sqs:GetQueueAttributes`, `ec2:DescribeNatGateways` | the inventory list reads behind the thresholds | `ReadOnlyAccess` (not `CloudWatchReadOnlyAccess`, which grants none of them); the CloudShell read role (its own region); the platform's inline read policy names them |
| `health:DescribeEvents` | provider health (R37); without a Business, Enterprise On-Ramp or Enterprise support plan the public AWS Health Dashboard instead | `ReadOnlyAccess`; the CloudShell read role since 0.1.144 (on the global endpoint, signed in `us-east-1`) |
| `ec2:DescribeInstanceStatus`, `rds:DescribePendingMaintenanceActions` | scheduled maintenance (R35, R36) | `ReadOnlyAccess`; the CloudShell read role (its own region: `rds:DescribePendingMaintenanceActions` since 0.1.153, re-run the setup for an older role); the platform's inline read policy names them |
| `apigateway:GET` on `arn:aws:apigateway:<region>::/restapis` and `…::/apis` | REST and HTTP API names | the CloudShell read role grants these two collections only. The platform's Terraform read role carries `ReadOnlyAccess`, which grants `apigateway:GET` on every path, API key values included, so its inline policy **denies** the key paths (`/apikeys`, a usage plan's `/keys`); the agent also pauses for approval before it reads them, whatever role you supplied |

The CloudShell read role lists Lambda functions (`lambda:ListFunctions`, since 0.1.146). A page
carries each function's environment variables: values under a secret-looking name (`DB_PASSWORD`,
`API_TOKEN`) are masked before anyone or any model sees the result, the others are kept.

A missing permission is never a silent zero. The connection's line on Datasources and the card
say **needs cloudwatch:GetMetricData** (or `ListMetrics`) for the account and region when AWS
answers *access denied*. Refused credentials — deactivated or expired keys, an expired
session, a clock out of step — read as *the connection's credentials were refused (code)*
instead, and a role that cannot be assumed says so: there is no permission to add. Each is
retried once an hour, and at once after a passing **Test** or a configuration change. When every always-published series
of an account (running instances' CPU, for example) comes back empty, the card says
**evaluation returned no data** instead of reporting the account healthy, and a listing cut
at its page cap says **listing truncated**.

## Which connection evaluates which account

Each (account, region) is evaluated once, through one connection: the account's inventory
connection when its regions cover that region, otherwise the first connection by name that
runs **Built-in detection** and covers it. A CloudShell read role is pinned to one region, so
several connections into one account usually split the regions between them. The job is one of
the **Jobs this connection runs** on the connection (config key `builtin_detection`; only an
explicit `false` opts a connection out). Several workspaces on one shared account evaluate,
and pay, separately; a workspace with **workspace scope** tags evaluates only the resources
carrying all of them (an untagged resource is outside the scope), a connection with **Only
findings on resources tagged** only the resources it harvested that carry that tag (exact
value), and either never shows the "no inventory resource" count, which would reveal other
applications' resources.

## Cost

GetMetricData costs **US$ 0.01 per 1,000 metrics requested**, billed to the AWS account the
connection reads. The card counts the metrics each poll requests, the totals for today and
yesterday, and the estimated month per account at the current poll and coverage. Example:
300 resources × 3 metrics, every 5 minutes, is about 7.8 million metrics a month, about
US$ 78 — the same order as 900 standard alarms. `ListMetrics` calls (about one per metric
per region per hour) fall inside CloudWatch's free API requests for most accounts.

**Max resources per poll** (300 by default) caps the read; the cap is filled round-robin
across kinds, so a fleet of instances never starves the databases and load balancers, and a
resource with a rule in alarm is always read. What the cap leaves out is counted as not covered.

## Noise controls

- **M-of-N** on 1-minute datapoints (EC2 instance metrics: 5-minute), as in the table.
- **An evaluation delay of 600 s:** the newest datapoint read is 10 minutes old, so data
  CloudWatch has not delivered yet is not read as missing (Datadog's CloudWatch monitors use
  900 s).
- **Missing data never breaches.** A rate needs at least **10 requests (or invocations) a
  minute** to count. A rate already in alarm keeps its state while traffic runs under that
  floor (no evidence either way), so a quiet night never clears a load balancer still failing
  every request; traffic that stops altogether is missing data and clears it.
- **Clear after three clean polls in a row.** A failed read is not a recovery: the state stays.
- **A resolve from elsewhere never silences a rule still in alarm.** Alerts resolve by
  fingerprint, whoever sends the resolve; a built-in alert closed that way while its rule is
  still breaching is sent again at the next poll and rejoins (or reopens) its incident.
- **Warning rules are held.** A warning opens or joins its incident with no investigation, no
  ticket, no push and no paid recheck (the recheck answers from the alert rows at no cost).
  **Analyse** on the incident, or a critical rule (or any other alert) joining it, starts the
  investigation and files the ticket as an open would. A workspace can let warning rules start
  an analysis (**Warning rules start an analysis** on the card).
- **Grouping and correlation** apply as to any alert. A held incident takes nobody in, held
  warnings buy no grouping turn, and the correlator sweep skips held built-in incidents. What
  was already in alarm when the workspace armed itself is a backlog of unrelated resources
  sent together, not a burst: it is never folded by shared cause (one incident per resource).
- **Flapping:** with a reopen window set, a held warning that clears and fires again inside it
  reopens the same incident (held again) instead of a new one per episode.

## Shadow mode, and arming

Turning detection on starts a **shadow period** (24 hours by default): rules are evaluated and
every pair that *would have fired* is recorded on the card with its observed value and, per
rule, a hit total; nothing is opened. The shadow clock runs only while something is covered —
with no inventory yet, a missing permission or every connection opted out it pauses, and the
card says why.

When the period ends the workspace **arms itself**. Pairs already in alarm at that moment open
their incidents with **no analysis** ("in alarm before built-in detection armed"), because
nobody was watching: a critical one still files its ticket and sends the open notification
(press **Analyse** to start the analysis), a warning stays held like any warning. New episodes
after that follow the usual severity rules. The owner can **Arm now** (the confirmation says how
many incidents open and how many start an analysis; what is already in alarm then follows the
usual rules), **Keep shadow** indefinitely, or switch single rules off in the rule table. Arm
now takes effect at the next poll — the card says it is waiting until then — and is not offered
while Keep shadow is on (untick it first); ticking Keep shadow cancels an Arm now still waiting.

## Turning it on and off

- **Workspace owner** (`members.manage`): the card's **Turn on** / **Turn off**. Turning off
  resolves every built-in alert still active, annotated *built-in detection switched off — no
  longer watched*; the confirmation lists the incidents first (their titles to a role that may
  read incidents, else their number). Turning on again starts a new
  shadow period.
- **Per connection:** untick **Built-in detection** under *Jobs this connection runs*. When no
  other connection evaluates that account and region, its built-in incidents resolve at the
  next poll with the same kind of note; the dialog lists them before you save.
- **Stopping to watch is not a clear** (turning off, a rule switched off, an opted-out
  connection, a resource leaving the workspace scope or the connection's tag scope): the
  incidents resolve with a note saying
  nothing was seen to clear, no recovery time is recorded, no verification runs after the close,
  and a ticket the incident has gets the write-back but **stays open** for its owner.
- **Platform staff:** the ops console's **Built-in detection (AWS)** switch stops it for every
  workspace. It freezes rather than resolves: alerts already firing stay open until it is back
  on, and the card says so. Disabling the AWS connector type itself freezes it the same way.
- A pair in alarm that no poll could read for 24 hours (a lost permission, an unreachable
  account) is listed on the card; it is never resolved by a failed read.

## What an alert looks like, and how it sits beside your own alarms

Each alert carries `alertname` = rule ID and name (`R11 ALB 5xx rate`), `service` = the
resource's name in the inventory, `resource` and `resource_id` = its ARN, `resource_type`,
`rule`, `severity`, `cloud=aws`, `account`, `region`, `metric_namespace`, and the CloudWatch
dimensions as labels (`LoadBalancer=app/web-alb/…`), the way the CloudWatch alarm webhook
labels an alarm. The incident title reads *Built-in rule R11 · ALB 5xx rate > 5 % for 5 of
5 min · web-alb*; the alert says what it observed against the threshold and links the rule's
source. Target-group rules key on the target group itself, so one load-balancer outage can
open an ALB incident and a target-group incident.

Beside a native CloudWatch alarm on the same resource: the alarm webhook names the resource
by its dimension value, a built-in alert by its name in the inventory. They agree for an EC2
instance, an RDS instance and a Lambda function (the inventory names those by their ID or
name), and for an SQS queue or DynamoDB table **without** a `Name` tag: the two alerts land on
one incident. An EFS file system, or a queue or table with a `Name` tag, is named after that tag
in the inventory (`shared-data` against `fs-0abc…`); load balancers, target groups, OpenSearch
domains and API Gateway stages are named differently too (`web-alb` against `app/web-alb/<id>`).
Those open separate incidents unless the alarm's description declares `service=<name>`. Both
carry the dimension labels, so a `label_map`, `criticality_map` or `assignment_map` entry can
target either. The source `builtin` is the platform's own: a webhook sender that names it is
recorded as `generic`.

## Limits

- **Latency:** the 10-minute delay, plus the rule's window, plus up to one poll (5 minutes)
  from the first bad minute to an alert: about 11–16 minutes for a 1-of-1 × 1 min rule, 15–20
  for 5 of 5 × 1 min (or 1 of 1 × 5 min), 20–25 for 10 of 10 × 1 min or 2 of 2 × 5 min, and
  25–30 for 15 of 15 × 1 min or 3 of 3 × 5 min. About 25 minutes to clear. A native alarm is
  about 5 minutes.
- **Low traffic:** a rate needs 10 a minute; a quiet load balancer or function failing every
  request never fires.
- **An NLB target group with no registered targets** is not seen (its health metrics are not
  published). An ALB target group that lost every target is seen through R11, because the
  load balancer's own 503s count.
- **Inventory:** SQS retention is read for
  20 queues per region per harvest, a different 20 each day, and what was read is kept (R18
  reaches every queue after a few daily harvests); a REST API's rules need its name, from the
  list read. On a large account the per-harvest call cap reaches these list reads last, after
  every region's existing estate reads; the harvest notes say what it skipped. The first harvest after this release shows every RDS instance and every resource the
  new list reads describe as changed or new.
- **Baselines** exist for latency and fault rate (R26 to R28) only. Not in this phase: error
  bursts and traffic drops against a baseline, the further rules X1–X8, Application Signals
  dependency (caller-side) metrics, services outside EKS, and logs. A service that Application
  Signals sees only from its callers (no server-side instrumentation) has no service-level series
  for R26 and R27 to read.


## Next steps

- Next: [Synthetic checks](/help/synthetic-checks.md)
- [All Connect your environment pages](/help/connect-your-environment/llms.txt)
- [Documentation home](/help/)