Notifications, paging, on-call and tickets — set it up like the demo

Send incident events to chat, email and pagers, set the on-call schedule, and file and write back tickets.

How-to · Updated 2026-10-26 · Describes release 0.1.164

Goal

Connect a controlled notification destination and, if required, a ticket system. Verify the sending audit and the receiving system independently. A webhook receiver shows that a notification was sent; it is not a PagerDuty or Opsgenie page.

Prerequisites

Procedure

  1. Open Settings · Account → Notifications & on-call → Incident notifications — outbound. Add the intended target kind: Teams, Slack, webhook, email, PagerDuty or Opsgenie. Select only the events you need: open, escalate and resolve, and on a chat, email or webhook target also golden (the daily agent check failed), rollback (a change failed, and what happened to its rollback) and image_not_deployed (an image the platform rebuilt under a new tag whose old digest the scanner still reports — sent when the validation finds it and again every 24 hours, aiops_image_reminder_h, until the scanner stops reporting the old digest; moving the workloads alone does not stop it: delete or untag the old digest once nothing runs it) and change_schedule (a compliance change scheduled to run later, then released or not run at its time, or cancelled). A target that lists its events gets image_not_deployed and change_schedule only once you tick them. Enter credentials through the private setup flow. You should see a saved target with its intended events; secrets must not reappear in evidence.

    API: PATCH /api/aiops/{tid}/guardrails with notifications — members.manage. An owner read can include a webhook URL, so use a sanitized receipt for shared evidence.

Notifications & on-call → Incident notifications — outbound: a Slack target with open, escalate and resolve ticked and golden and rollback offered, the message interval left on the platform default, Add target and Save notifications.
Notifications & on-call → Incident notifications — outbound: a Slack target with open, escalate and resolve ticked and golden and rollback offered, the message interval left on the platform default, Add target and Save notifications.

  1. For PagerDuty, use a PagerDuty routing key (Events API v2 integration key); for Opsgenie, use its API key. Choose the appropriate US/EU region. Key stored — leave blank to keep it means the key is write-only. You should see the target/key-presence state, never a key value shown back.

    API: a pager is an entry of the same notifications list — kind pagerduty or opsgenie, region us or eu, a write-only key and its events. One incident identity is reused as the pager deduplication identity; escalation/reopen re-trigger it and closure resolves it. Generic webhook delivery has different semantics.

  2. Set Message interval (seconds) only if digesting chat/email messages is desired. Blank follows the platform default. You should see the effective interval and the recorded delivery behavior, rather than assuming every state transition produces a separate message.

    API: PATCH /api/aiops/{tid}/guardrails with notify_min_interval_s — members.manage.

  3. Review Email delivery and On-call schedule. Use one member per line, the intended daily/weekly rotation, anchor date and overrides. Enable on-call routing on the relevant email target. You should see the configured recipient schedule and the actual receiver selected for the test time.

    API: PATCH /api/aiops/{tid}/guardrails with oncall or smtp — members.manage. A configured relay is not proof that a message was delivered.

Notifications & on-call → On-call schedule and Email delivery: who is on call now, the rotation list and anchor date, and the workspace SMTP relay whose password is write-only.
Notifications & on-call → On-call schedule and Email delivery: who is on call now, the rotation list and anchor date, and the workspace SMTP relay whose password is write-only.

  1. Deliver one approved test signal and inspect both the incident Timeline and Audit log entry incident.notified and the target's receiving record. Resolve the test through its source and verify the relevant recovery event. You should see the actual event, target kind and delivery result paired with a receiver receipt.

    API: audit reads use GET /api/aiops/{tid}/audit — report.view. The approval gate is never a notification event; review Needs you or the pending-approval API. A PagerDuty or Opsgenie target takes open, escalate and resolve only: switching a row to a pager clears golden, rollback, image_not_deployed and change_schedule, and the API refuses them on a pager.

Incident page, Timeline: every notification sent or held for the digest is a row with its target kind and event, in order with the status changes and the fix links.
Incident page, Timeline: every notification sent or held for the digest is a row with its target kind and event, in order with the status changes and the fix links.

  1. On the ticket-system connection, open Tickets, fill its scope, and Check before saving. The check is read-only. You should see the discovered/project choices and the actual access result.

    API: POST /api/aiops/{tid}/datasources/{id}/ticket-discovery — datasources.manage.

Data sources → Tickets on a Jira connection: the project key, the read-only Check, the note on which incidents the route files, and Save ticket route, disabled until the check has run.
Data sources → Tickets on a Jira connection: the project key, the read-only Check, the note on which incidents the route files, and Save ticket route, disabled until the check has run.

  1. Understand the route before Save ticket route. The ordinary dialog saves a catch-all route: no selector, so it files a ticket for every incident of the incident pillar — a preset covers only the pillars its pillars parameter names, the incident pillar by default; another save updates it. On a busy workspace, use a deliberately scoped route instead of unintentionally enabling ticket creation for everything. Send the approved test ticket after setup and inspect it in the target. You should see one scoped route and a test result for the destination you intended.

    API: POST /api/aiops/{tid}/routes or PATCH /api/aiops/{tid}/routes/{id} — datasources.manage; test via POST /api/aiops/{tid}/routes/{id}/test-ticket. Advanced routes specify a selector.

    A fixed ticket-system preset needs no route certification. A new or edited hand-written on open or on resolve API route does: an approver reviews and certifies its current revision before it can send requests automatically. Such a route can target only a ticket, paging or webhook system through a read connection, never an estate-remediation writer. An older migrated pre-approval can show certifier unknown: it records a pre-approval from before certification existed, not a named person's certification. Read Needs you → Certify and the route’s actual certification state if it is held; changing its content drops the previous certification. See runbooks and automatic fixes.

Legacy remediation routes: the hand-written on open / on resolve ticket routes carry certifier unknown; Certify opens the review an approver signs.
Legacy remediation routes: the hand-written on open / on resolve ticket routes carry certifier unknown; Certify opens the review an approver signs.

  1. Follow a real incident's L3 Incident creation & routing ticket link, then its write-back and close receipt. Held/unmapped cases wait until Analyse before filing. Inspect Ticket filing failed, or the recorded write-back failure, and use Retry only against the actual failed operation. You should see one ticket for the incident and its recorded updates, rather than a second ticket for the same incident.

  2. If tickets are also inbound signals, map the service identity so the ticket joins the existing incident. An adopted customer-owned ticket keeps its status; Write back is explicit. ServiceNow correlation fields must follow the integration recipe to avoid an echo loop. You should see the adopted reference and source provenance.

    API: the source-specific Jira/ServiceNow/Azure DevOps webhook uses workspace webhook authentication. The platform marks the tickets it files — Jira labels aiops and aiops-<incident id>, ServiceNow correlation_id aiops:<incident id> with correlation_display AIOps — and drops its own tickets when they come back through the webhook, counted as ticket_echo in the response, so no vendor-side filter is needed; a ServiceNow inbound business rule must post correlation_id and correlation_display with the record, or the platform cannot tell its own ticket from a customer's while its create is still in flight. The payload fields and the business rule itself are in Ticket systems → Inbound webhooks and echoes.

Verify and cleanup

Retain the source, platform notification/ticket audit and receiver receipts. Remove temporary targets and disable/remove only the temporary scoped route after the agreed test. Confirm the restored target/route state without exposing keys or URLs.

Troubleshooting

Symptom Check Next step
Pager reports no stored key Target kind, region and key-presence result Store the correct integration key through the private setup flow
Webhook received a message but no page exists Actual target kind It was a notification, not a page; configure a real pager to test paging
No ticket yet The reason L3 gives: held, unmapped or no route Resolve the named gate; do not assume connection existence enables filing
Write-back fails Recorded external response and adopted-ticket ownership Review the integration and its access; retry only the recorded operation
Approval seems missing from messages Approval gate is not a push event Poll Needs you or pending approvals

Alert sources · Ticket systems · Audit log → Notification setup checks · REST API.

Next steps