Automation in your account

A scheduled function with no AI in your AWS account probes the endpoints the platform cannot reach and posts your CloudTrail writes; the platform still does the fixing.

How-to · Updated 2026-10-28 · Describes release 0.1.165

The platform probes synthetic checks from where it runs, so an endpoint on a private address — an internal load balancer, a database port, a service inside a VPC — is out of its reach. The automation closes that gap without a runtime of your own: a small function that runs in your AWS account on a schedule, probes those endpoints and posts the results to the platform. It can also read your CloudTrail and post the writes it finds as changes.

It has no AI in it and decides nothing. A result lands on its check exactly like a probe from the platform: a failure confirmed by one retry opens or joins an incident at once, two passes in a row close it, a suggestion in trial opens nothing. The automation writes nothing in your account except its own table; it creates no CloudWatch alarm. The platform investigates and fixes through its own connection, as before. The code is plain Python you can read: deploy/aws/automation/index.py in the platform repository.

Where it lives: Settings · Data in → Synthetic checks (in the product: /portal/settings/monitors), card Automation in your account.

Before you start

Create the token

  1. Open Synthetic checks. The card Automation in your account reads Not set up.
  2. Select Create token. Copy the token at once: it is shown once and only its hash is kept. The card also shows the Automation endpoint the automation calls.
  3. Rotate token mints a new one and stops the old one at once — the installed automation is refused until its secret holds the new token. Revoke token stops it without a replacement.

You should see Token created — the automation has not called in yet. until the automation's first call, then Live — last called in ….

Install it

The guided CloudShell setup installs the automation in the same reviewed CloudFormation stack as the connection roles.

  1. Open Settings · Data in → Keyless cloud access (in the product: /portal/connect/keyless), choose AWS, the account and the region, and tick Install the automation in this account.
  2. Choose how often it runs (Run every: 1, 5, 10 or 15 minutes, 5 by default) and whether it should Post CloudTrail writes as changes (on by default). The event names it posts follow a regular expression you may change; empty means names starting with Create, Update, Modify, Delete, Put, Attach, Detach, Associate, Disassociate, Authorize, Revoke, Start, Stop, Reboot, Terminate, Run, Enable, Disable, Register, Deregister, Add, Remove, Replace, Restore or Set.
  3. To reach private endpoints, give the VPC ID and its Subnet IDs. Without your own security groups the setup creates one that allows outbound HTTPS (port 443) only and nothing inbound, and tells the function so: a private check on another port (a database, port 80 of an internal load balancer) is not handed to it and keeps Waiting for the automation in your account — give your own security groups to probe those. The subnets need a route to the platform, and to Secrets Manager, DynamoDB and CloudTrail in the stack's region and in us-east-1 — a NAT gateway, or a private connection to the platform plus endpoints for those services.
  4. Optionally forward the Security Hub findings of this account and region to the workspace's securityhub webhook: an EventBridge rule sends the Security Hub Findings - Imported events through an API destination — every new finding, every change, and every clear, but not a control re-check that stayed passed (Security Hub re-sends each of those on every check). It sends at most 5 a second and retries a refused delivery for an hour at most. This one uses the workspace webhook token, not the automation token. The workspace's scanner pull reads the same findings when it is on; either one alone is enough. API destinations exist only in some regions; the setup refuses the option elsewhere.
  5. Select Create automation token (or take the one you created on Synthetic checks), then Preview setup command, run it in CloudShell and review the change set. When the workspace already has a token, the setup offers Replace the automation token instead: paste the token you saved, because a new one stops the automation installed with the old one. The command asks for the automation token, and for the webhook token when forwarding Security Hub; they are typed, never shown and never on a command line. On a later run an empty answer keeps the stored token.

You should see the card on Synthetic checks turn Live within one schedule interval.

The stack adds: the function (Python 3.12, its code inline in the template so you can read it there), its execution role, a DynamoDB table (on-demand), a Secrets Manager secret holding the automation token, a log group kept 30 days and the schedule. The role may read and write only its own table item, read only its own secret, write only its own logs and — with CloudTrail on — call cloudtrail:LookupEvents in the stack's region and us-east-1; in a VPC also the network-interface actions Lambda needs, denied to the function's own code. The setup refuses a change set that removes or replaces anything, so turning the automation off is not a setup run: revoke its token (it is refused at once) and disable its schedule rule in EventBridge.

Which checks it runs

A check probed by the automation reads … from your account after its state. Each check keeps its own interval, but the automation runs on its schedule: a check is probed at most once per run, so its interval is never shorter than the schedule. The platform keeps the cadence too: of one post it takes only the newest result per check, and it ignores a result that comes sooner than half the check's interval after the last one, so a retried or repeated post cannot open an incident each time.

One automation per workspace

Every call names the function that makes it. One automation runs per workspace — one list of checks, one heartbeat — so while it is Live, another one (a second account, a second region) is refused, and the card names it: Another automation (…) was refused. Install it in the one account whose private endpoints you need; to move it, remove it and install it elsewhere, and the new one takes over once the old one has been silent for 30 minutes.

The function that evaluates the built-in rules in your account (below) is a second function with its own record and heartbeat, so it runs beside this automation, never instead of it. The same rule holds for it: one per workspace, and a second one is refused while the first is live.

Alive or stale

Every run the automation asks for its checks and posts its results — an empty list too. The results post is the heartbeat: a run that dies before it does not count. The card reads Live while a run finished within the last 30 minutes, Stale after that. A stale automation probes nothing, so the platform no longer knows how its checks stand: they read Unknown — the automation in your account has not finished a run since … (an incident already open stays open), and one alert, Automation silent, opens an incident while the token is set and it runs an armed check. Its next finished run resolves it. A revoked token is a decision, not an outage: its checks read Unknown, with no alert. A check cannot be moved to the automation while no token is set.

Changes from your CloudTrail

With CloudTrail on, the automation reads your account's management write events (LookupEvents with ReadOnly=false, in its own region and us-east-1, where the global services such as IAM write) every 5, 10 or 15 minutes, and posts the ones whose event name matches its pattern to the change webhook — in batches of 100, each event once. It skips a failed call, an AWS service acting on its own, and the writes of the platform's own roles (the platform records those as its own changes); the platform drops what its own read calls noise (session plumbing such as log streams and CloudShell sessions, the Systems Manager agent's reports, Config verdicts, a service's own reconcile loop). Each lands in the change ledger like any change a webhook sends: it answers "what changed on this service" and the freeze-window check, and the card says when writes were last posted. The read overlaps the previous one by 10 minutes, so an event that shows up in the event history after the read that should have seen it is still read, once; a read that fails in the stack's region is retried at the next run from the same point (us-east-1 failing leaves it out of that read, as the platform's own read does). A busy window — more than 2,000 writes in one region in one read — is read newest first; the next run reads the rest before it moves on. A post the platform refuses for rate waits for its Retry-After; the batches that landed before it are never posted again.

The platform's own CloudTrail read through your AWS connection carries on: the same event read by both is one change, matched on the CloudTrail event id. When the platform's read reaches an event the automation posted first, it takes the row over and judges it as its own — who made it (the principal check), whether it is the echo of a platform change, the reference check for an unauthorized change — and the estate inventory follows it. Until then the row is the automation's report, marked as reported by a sender.

Read CloudTrail less often on the platform side

The platform's own CloudTrail read through your AWS connection runs every 15 minutes (aiops_change_harvest_interval_s). A workspace whose automation already posts the writes may relax it: with the workspace setting aiops_automation_relax_harvest set to true (off by default), the platform reads CloudTrail itself only every hour (AIOPS_AUTOMATION_RELAXED_HARVEST_S, 3600 seconds) for the AWS connection of the automation's own account whose regions the automation reads, while the automation is Live and its last complete CloudTrail read is less than 30 minutes old. Any other connection — another account, a region the automation does not read, a us-east-1 it could not reach — keeps the usual interval, as does every connection when the automation goes Stale or stops reading. Only the AWS CloudTrail read relaxes: Azure, the repositories and "pull now" keep theirs.

The trade-off: the automation's row is a sender's report. Until the platform's own read reaches the same event — up to an hour later — the change has no principal check (who made it, and whether it was the platform's own credential), no match to a platform change it echoes, no reference check for an unauthorized change, no before-state from AWS Config, and the estate inventory does not follow it yet. Keep the switch off where those must come within the usual 15 minutes; turn it on, to cut the platform's CloudTrail requests, where they may wait up to an hour.

Built-in rules evaluated in your account

The platform's built-in detection rules (37 today) are also served as data, the rule dictionary, for a function in your account to evaluate there with GetMetricData and to signal only on a breach; the platform never sends code. While a results post says its run evaluated the dictionary for every reader built-in detection runs, in the account and every region of each AWS connection that runs detection, the platform's own detection poll of the workspace pauses. It resumes at the next scheduler tick once that run is more than 15 minutes old, or as soon as that function posts without rules. Only that function's runs count; the synthetic-check automation never pauses the poll. Another account, a region it does not read, a reader it does not run, a Kubernetes connection to a cluster it did not read or a revoked automation token keeps the platform polling the whole workspace; so does a run that used a cached or baked dictionary or left an alert undelivered, since it posts without rules.

What it costs

The cost is yours: the function runs in your account. Plan for about USD 15 a month for the function, its table, secret and logs with the CloudTrail scan every 10 minutes (the default; 5 or 15 through the API); in a VPC without internet add a NAT gateway or interface endpoints for CloudTrail and Secrets Manager, which cost more than the function itself (Reach the platform privately). What follows is an estimate of volume, not a measured bill: price it with the AWS pricing pages for Lambda, DynamoDB, Secrets Manager and CloudWatch Logs. At the default 5-minute schedule the function runs about 8,640 times a month (43,200 at 1 minute); each run lasts about as long as its slowest probe (a failing probe waits for its timeout, 10 seconds more and its retry), makes one or two DynamoDB reads and as many writes, reads its secret only when it starts cold and writes a few log lines. One secret is stored. The CloudTrail read adds up to 2 lookup requests a second while it pages, a few times an hour.

API

The automation calls these endpoints with Authorization: Bearer <automation token>:

Each call carries the function's ARN in X-Aiops-Automation-Instance; another function of the same kind is refused with 409 while this one is live. They have their own intake rate bucket, the size of the workspace's alert intake limit, so they never use up the alerts', and the 1 MiB body limit of every webhook. The token itself: POST and DELETE /api/aiops/{tenant_id}/monitor-automation/token (MCP mint_automation_token, revoke_automation_token); PATCH …/monitors/{check_id} with {"prober": "automation"} or "platform" moves a check.

Next steps