Report changes to the change webhook

Report deploys, releases and settings to the change ledger from your own tools, Kubernetes audit logs or Argo CD, and choose a git watch list, a webhook or both.

How-to · Updated 2026-10-26 · Describes release 0.1.164

Before you start. Every change endpoint takes the workspace's webhook token, sent as for alerts (Create the webhook token), except the Kubernetes audit webhook, which takes its cluster's own audit token (below). A change never opens an incident: it lands on the change ledger, ranks on an incident's Probable cause card and is flagged when it falls in a freeze window. Git hosts, ticket systems and cloud audit logs report through their connections and the endpoints in Change sources; this page covers the sources you wire yourself and what the card needs to be useful.

Report changes to the change webhook

A change made outside the platform's view — a deploy tool, a release script, a setting saved in an application's own admin screen — can be reported to POST /webhook/change/<tenant-slug> with the webhook token, sent as for alerts. Events land on the change ledger, never as alerts, and the answer is {"recorded": N, "flagged_unauthorized": M}.

{"source": "release-bot",
 "events": [{"kind": "deploy", "service": "checkout", "sha": "9f1c2ab",
             "title": "checkout 2.14.0", "at": "2026-10-01T09:05:00Z"}]}

kind is push, pr_merged, deploy, sync or config. Give service the value your alerts carry in their service label, so an incident's Probable cause card can relate the change to the failing service.

A setting changed inside an application is kind: "config", with service naming the application and detail the setting:

{"source": "billing-admin",
 "events": [{"kind": "config", "id": "audit-4711", "service": "billing",
             "title": "retry limit lowered", "at": "2026-10-01T09:12:00Z",
             "detail": {"key": "payments.retry_limit", "old": "5", "new": "1"}}]}

It is listed as a reported version — evidence, never a restore source. id is the application's own audit-row id; events dedupe on the source, the service and the id.

Never send a secret. A config event's values are protected by the setting's name only: under a key that looks secret (a word such as password, secret, token, key, pass, pwd, dsn, auth, cert, salt or bearer) only key, resource_id and service are kept and old and new read [not stored], and a URL's password (scheme://user:password@host) reads [not stored] in every detail value. A secret under an innocent name, or in the title, is kept as sent after the PII scrub — and every other event kind is stored as sent after the PII scrub, a URL's password included. Never put a credential in a change-webhook payload.

The event fields and the 400 for an empty list are in Your own change feed.

Kubernetes on-prem: the kube-apiserver audit webhook

For clusters where you control the API server flags (kubeadm, RKE, k3s). EKS has no audit webhook — use the connection key below; AKS and GKE ship audit logs to Azure Monitor / Cloud Logging, which the platform does not read yet (planned).

The webhook config is a kubeconfig; its token: sends the cluster's audit token as Authorization: Bearer, so it never sits in the URL. Each cluster has its own: POST /api/aiops/{workspace}/k8s-audit-token/<cluster-name> (pillars.manage, MCP mint_k8s_audit_token; audit tenant.k8s_audit_token) returns it once and stores only its hash, and minting again for the same cluster replaces it. A personal access token needs the admin scope for this call. The workspace webhook token is refused on this path: what the API server posts is recorded as the platform's own audit trail (a refused write is a denied attempt, and a later copy of the same audit event is a duplicate), which only the cluster's own token may write. The policy records write verbs only, at Metadata level — no request bodies, a few events per human action. /<cluster-name> is the service of every row; use the name your alerts carry for that cluster.

# kube-apiserver flags
--audit-policy-file=/etc/kubernetes/audit/policy.yaml
--audit-webhook-config-file=/etc/kubernetes/audit/webhook.yaml
--audit-webhook-batch-max-wait=30s

# /etc/kubernetes/audit/webhook.yaml
apiVersion: v1
kind: Config
clusters:
- name: aiops
  cluster:
    server: https://<platform>/webhook/kubernetes-audit/<tenant-slug>/-/<cluster-name>
users:
- name: aiops
  user:
    token: <cluster-audit-token>
contexts:
- name: aiops
  context: {cluster: aiops, user: aiops}
current-context: aiops

# /etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: [RequestReceived, ResponseStarted, Panic]
rules:
- level: None
  users: ["system:kube-controller-manager", "system:kube-scheduler", "system:apiserver"]
- level: None
  userGroups: ["system:nodes", "system:serviceaccounts:kube-system"]
- level: Metadata
  verbs: [create, update, patch, delete, deletecollection]
- level: None

The API server posts events in batches (--audit-webhook-batch-max-size, 400 events by default). This path accepts up to 16 MiB per POST and answers 413 above that. If ingress-nginx is in front of the platform, its proxy-body-size default of 1 MiB applies first: raise it on the platform's ingress (nginx.ingress.kubernetes.io/proxy-body-size: 16m) or lower the batch size.

The - is the token's place in the path: the token itself rides in the kubeconfig token: (sent as Authorization: Bearer) and the last segment names the cluster — the API server's webhook client drops any query string, so the name has to be a path segment. The token opens only the cluster it was minted for; a wrong token, cluster name or workspace slug all answer the same 401.

Upgrading from 0.1.155 or earlier. Before 0.1.156 this path took the workspace webhook token, and /webhook/kubernetes-audit/<workspace-slug>?cluster=<cluster-name> was accepted too. Both are refused now (401 on the cluster path, 404 on the form without the cluster segment), and the API server drops the events it cannot deliver. For each cluster, mint its audit token (POST /api/aiops/{workspace}/k8s-audit-token/<cluster-name> with an admin-scope personal access token, or MCP mint_k8s_audit_token; the portal has no control for it), put it in the kubeconfig's token: and restart the kube-apiserver; the writes made in between are not recorded. The webhook token itself is unchanged for every other source.

Of the EventList the API server posts, a ResponseComplete create / update / patch / delete / deletecollection by a user not starting with system: becomes one change row (who, verb, resource, name, namespace, when, cluster); a 401 or 403 on the same is recorded as a denied attempt; everything else is dropped. The Payload template for kubernetes-audit on the Webhook ingest card carries the same three files with your workspace filled in.

Kubernetes on EKS: k8s_audit: true on the AWS connection

No webhook and nothing installed in the cluster. Turn on the audit control-plane log type on each cluster, then add "k8s_audit": true to the AWS read connection's config, next to config_history: true. The scheduled change harvest (every 15 minutes by default, alongside the CloudTrail pull) then reads /aws/eks/<cluster>/cluster, streams kube-apiserver-audit-*, through CloudWatch Logs Insights for every EKS cluster in the connection's regions that has audit logging on — the Kubernetes API server is never read. The read role needs logs:DescribeLogGroups, logs:StartQuery, logs:GetQueryResults and eks:DescribeCluster. Writes by system:serviceaccount:… identities are excluded: a GitOps controller's write is the authorized path and arrives as an Argo CD sync event instead, so a human kubectl apply stands out.

Argo CD drift and sync

The argocd endpoint reads what argocd-notifications posts — {"app", "revision", "status", "url", "at"} — and records one sync row per Application and revision, named after the Application, plus a separate drift row (argocd drift <app>: live state differs from git) when the notification reports OutOfSync for that revision. Add the service, template and triggers to argocd-notifications-cm, the token to argocd-notifications-secret, and one annotation per trigger on each Application:

# argocd-notifications-cm (data:)
service.webhook.aiops: |
  url: https://<platform>/webhook/argocd/<tenant-slug>
  headers:
  - name: X-Webhook-Token
    value: $aiops-webhook-token
  - name: Content-Type
    value: application/json
template.aiops-sync: |
  webhook:
    aiops:
      method: POST
      body: |
        {"app": "{{.app.metadata.name}}", "revision": "{{.app.status.sync.revision}}", "status": "{{.app.status.sync.status}}", "url": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}", "at": "{{.app.status.reconciledAt}}"}
trigger.on-out-of-sync: |
  - when: app.status.sync.status == 'OutOfSync'
    oncePer: app.status.sync.revision
    send: [aiops-sync]
trigger.on-sync-succeeded: |
  - when: app.status.operationState.phase in ['Succeeded']
    send: [aiops-sync]
trigger.on-sync-failed: |
  - when: app.status.operationState.phase in ['Error', 'Failed']
    send: [aiops-sync]
trigger.on-deployed: |
  - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
    oncePer: app.status.operationState.syncResult.revision
    send: [aiops-sync]

# argocd-notifications-secret (stringData:)
aiops-webhook-token: <webhook-token>

# on each Application (metadata.annotations:), one line per trigger
notifications.argoproj.io/subscribe.on-out-of-sync.aiops: ""
notifications.argoproj.io/subscribe.on-sync-succeeded.aiops: ""
notifications.argoproj.io/subscribe.on-sync-failed.aiops: ""
notifications.argoproj.io/subscribe.on-deployed.aiops: ""

An OutOfSync row is the drift signal; the Synced row that follows is the sync. Redefining the four triggers replaces the catalog's versions of the same names — keep their send: lists next to aiops-sync if other channels already use them.

Which repositories a git connection watches

A GitHub, GitLab, Azure DevOps or Gitea connection also carries a Repositories field: the repositories the change agent polls for commits, comma- or newline-separated. Use owner/name — group/project on GitLab, <project>/<repo> on Azure DevOps. A pasted browser or clone address is read the same way: https://dev.azure.com/<org>/<project>/_git/<repo>, <project>/_git/<repo> and git@ssh.dev.azure.com:v3/<org>/<project>/<repo> all become <project>/<repo>, and https://github.com/acme/payments.git becomes acme/payments. A repository the last poll could not read shows on the Changes page as last poll failed with the HTTP status. Patterns such as acme/platform-* work on GitHub only; the other three need exact names. Twenty-five repositories per connection are polled each tick; anything past that is reported, never silently dropped. Only read-role, enabled connections are polled: repositories named on a write-role remediation credential are listed as named-but-not-polled.

Leaving the field empty is a real answer, not a default — the connection then watches no repository, and a push or merge in it is invisible to the change agent unless that repository sends a webhook. Changes states the answer per connection, so a provider that is connected but watching nothing is visible rather than silent, and suggests repositories the platform already knows about from your remediation routes, service map and estate x-ray. A suggestion is a hint to copy into this field; nothing is polled until you save it here.

The same suggestions are offered inside the connect dialog itself, beside the fields, so a repository can be added while the connection is being set up rather than on a later visit to Changes. Selecting one appends it to Repositories; it never replaces what you typed, and a repository already listed is shown as added rather than offered again. On a connection that will not be polled — disabled, or approved-changes-only — the dialog says so in place of the suggestions rather than inviting you to fill a field that does nothing.

Webhook or watch list, or both

The connect dialog's guidance column also gives the webhook setup for the provider being connected: the payload URL shape, the provider-side steps, and the token page to collect the actual token from. A webhook arrives immediately; the Repositories watch list is polling, so the same push lands minutes to hours later. They are complements, not alternatives — a repository can have both, and duplicate events are folded into one change record.

All four providers post their own native payload straight to this platform, so the endpoint shown in the dialog is the URL to paste into their webhook form — no relay and no payload rewriting. Select push and merged pull request (or merge request) events; a pull request closed without merging is not a change and is not recorded. A repository can have both a webhook and a place on the watch list: the webhook arrives immediately, the watch list is the safety net, and a commit seen twice is folded into one change record.

What the Probable cause card needs

An incident's Probable cause card can rank only the changes that were recorded, and look back only as far as the monitor's own history reaches. Five things make it useful:

  1. Alarms on the resources in scope, routed to the webhook. A CloudWatch alarm sends ALARM and OK to an SNS topic subscribed to /webhook/cloudwatch-sns/<slug>; an Azure Monitor alert rule's action group calls /webhook/azure-monitor/<slug>. Without an alert there is no incident, and the alarm's history is what Healthy since reads. On AWS with no alarms, Built-in detection opens the incident instead: its alert carries the rule's own datapoints, which give the error start (the first breaching minute) and the clean stretch before it.
  2. Change capture on. An AWS or Azure read connection whose audit log is harvested: the workspace harvest interval above 0 (every 15 minutes by default) and the connection not opted out with harvest_changes: false. Without it no pull runs when an incident opens, the alarm's history and the failing resource's attachments are not read either (Healthy since comes from the platform's own incident history), and the list holds only the changes the platform made and the webhooks reported.
  3. Before-states on AWS. config_history: true on the AWS read connection, and the AWS Config recorder on for the resource types in scope. Without them a change is listed with no before-state and cannot be reverted from the card. With them the before-state is the configuration Config captured at or before the write's own time — asked for by that time, so a resource rewritten several times after the write keeps it — and the first capture after the write gives the diff; a resource Config captured only after the write says so on its row. The look back reaches 28 days: a resource Config last captured longer before the write has no before-state.
  4. The alarm's account and region. The AWS read connection declares the region the alarm and its resources live in (only the declared regions plus us-east-1 are read for changes), and its role may read CloudWatch there; the bundled read roles grant cloudwatch:DescribeAlarmHistory. On Azure, Reader on the subscription covers the alert history. The alarm's history is read through the read connection that declares the alarm's AWS account (its role ARN or scope.accounts) or Azure subscription (subscription_id or scope.subscriptions), or through the only one when it declares none; otherwise it is not read and Healthy since falls back to the platform's own incident history.
  5. Lambda attachments. For a Lambda alarm the function's role, security groups and layers come from its AWS Config item, so the recorder must record Lambda functions and hold an item for the function from the last 28 days.

An alarm whose missing data counts as not breaching, or is ignored, can be OK on silence: the card never calls a change proven good on it.

Next steps