# Notifications, paging, on-call and tickets — set it up like the demo

> Send incident events to chat, email and pagers, set the on-call schedule, and file and write back tickets.

_How-to · Updated 2026-10-26 · Describes release 0.1.164_

## Goal

Connect a controlled notification destination and, if required, a ticket system. Verify the sending audit and the receiving system independently. A webhook receiver shows that a notification was sent; it is not a PagerDuty or Opsgenie page.

## Prerequisites

- `members.manage` for notification/on-call settings; `datasources.manage` for ticket discovery and routes.
- A destination owner, selected events, secret delivery channel, and a safe end-to-end test incident.
- For email, a configured SMTP relay (the platform default or an explicit workspace relay) and the workspace's delivery policy.
- For tickets, a supported Jira, ServiceNow or Azure DevOps connection with the required access.

## Procedure

1. Open **Settings · Account → Notifications & on-call → Incident notifications — outbound**. Add the intended target kind: Teams, Slack, webhook, email, PagerDuty or Opsgenie. Select only the events you need: open, escalate and resolve, and on a chat, email or webhook target also `golden` (the daily agent check failed), `rollback` (a change failed, and what happened to its rollback) and `image_not_deployed` (an image the platform [rebuilt under a new tag](/help/runbooks-and-automatic-fixes.md#image-rebuild) whose old digest the scanner still reports — sent when the validation finds it and again every 24 hours, `aiops_image_reminder_h`, until the scanner stops reporting the old digest; moving the workloads alone does not stop it: delete or untag the old digest once nothing runs it) and `change_schedule` (a compliance change [scheduled to run later](/help/change-control.md), then released or not run at its time, or cancelled). A target that lists its events gets `image_not_deployed` and `change_schedule` only once you tick them. Enter credentials through the private setup flow. **You should see** a saved target with its intended events; secrets must not reappear in evidence.

   API: `PATCH /api/aiops/{tid}/guardrails` with `notifications` — `members.manage`. An owner read can include a webhook URL, so use a sanitized receipt for shared evidence.

![Notifications & on-call → Incident notifications — outbound: a Slack target with open, escalate and resolve ticked and golden and rollback offered, the message interval left on the platform default, Add target and Save notifications.](/help/media/portal/portal-notifications.png?v=0.1.132)

2. For PagerDuty, use a **PagerDuty routing key (Events API v2 integration key)**; for Opsgenie, use its API key. Choose the appropriate US/EU region. **Key stored — leave blank to keep it** means the key is write-only. **You should see** the target/key-presence state, never a key value shown back.

   API: a pager is an entry of the same `notifications` list — `kind` `pagerduty` or `opsgenie`, `region` `us` or `eu`, a write-only `key` and its `events`. One incident identity is reused as the pager deduplication identity; escalation/reopen re-trigger it and closure resolves it. Generic webhook delivery has different semantics.

3. Set **Message interval (seconds)** only if digesting chat/email messages is desired. Blank follows the platform default. **You should see** the effective interval and the recorded delivery behavior, rather than assuming every state transition produces a separate message.

   API: `PATCH /api/aiops/{tid}/guardrails` with `notify_min_interval_s` — `members.manage`.

4. Review **Email delivery** and **On-call schedule**. Use one member per line, the intended daily/weekly rotation, anchor date and overrides. Enable on-call routing on the relevant email target. **You should see** the configured recipient schedule and the actual receiver selected for the test time.

   API: `PATCH /api/aiops/{tid}/guardrails` with `oncall` or `smtp` — `members.manage`. A configured relay is not proof that a message was delivered.

![Notifications & on-call → On-call schedule and Email delivery: who is on call now, the rotation list and anchor date, and the workspace SMTP relay whose password is write-only.](/help/media/portal/portal-notifications-oncall.png?v=0.1.132)

5. Deliver one approved test signal and inspect both the incident **Timeline** and **Audit log** entry `incident.notified` and the target's receiving record. Resolve the test through its source and verify the relevant recovery event. **You should see** the actual event, target kind and delivery result paired with a receiver receipt.

   API: audit reads use `GET /api/aiops/{tid}/audit` — `report.view`. The approval gate is never a notification event; review **Needs you** or the pending-approval API. A PagerDuty or Opsgenie target takes open, escalate and resolve only: switching a row to a pager clears `golden`, `rollback`, `image_not_deployed` and `change_schedule`, and the API refuses them on a pager.

![Incident page, Timeline: every notification sent or held for the digest is a row with its target kind and event, in order with the status changes and the fix links.](/help/media/portal/portal-incident-timeline.png?v=0.1.146)

6. On the ticket-system connection, open **Tickets**, fill its scope, and **Check** before saving. The check is read-only. **You should see** the discovered/project choices and the actual access result.

   API: `POST /api/aiops/{tid}/datasources/{id}/ticket-discovery` — `datasources.manage`.

![Data sources → Tickets on a Jira connection: the project key, the read-only Check, the note on which incidents the route files, and Save ticket route, disabled until the check has run.](/help/media/portal/portal-ticket-dialog.png?v=0.1.132)

7. Understand the route before **Save ticket route**. The ordinary dialog saves a catch-all route: no selector, so it files a ticket for every incident of the incident pillar — a preset covers only the pillars its `pillars` parameter names, the incident pillar by default; another save updates it. On a busy workspace, use a deliberately scoped route instead of unintentionally enabling ticket creation for everything. Send the approved test ticket after setup and inspect it in the target. **You should see** one scoped route and a test result for the destination you intended.

   API: `POST /api/aiops/{tid}/routes` or `PATCH /api/aiops/{tid}/routes/{id}` — `datasources.manage`; test via `POST /api/aiops/{tid}/routes/{id}/test-ticket`. Advanced routes specify a `selector`.

   A fixed ticket-system preset needs no route certification. A new or edited hand-written **on open** or **on resolve** API route does: an approver reviews and certifies its current revision before it can send requests automatically. Such a route can target only a ticket, paging or webhook system through a read connection, never an estate-remediation writer. An older migrated pre-approval can show **certifier unknown**: it records a pre-approval from before certification existed, not a named person's certification. Read **Needs you → Certify** and the route’s actual certification state if it is held; changing its content drops the previous certification. See [runbooks and automatic fixes](/help/runbooks-and-automatic-fixes.md).

![Legacy remediation routes: the hand-written on open / on resolve ticket routes carry certifier unknown; Certify opens the review an approver signs.](/help/media/portal/portal-legacy-routes.png?v=0.1.132)

8. Follow a real incident's L3 **Incident creation & routing** ticket link, then its write-back and close receipt. Held/unmapped cases wait until **Analyse** before filing. Inspect **Ticket filing failed**, or the recorded write-back failure, and use **Retry** only against the actual failed operation. **You should see** one ticket for the incident and its recorded updates, rather than a second ticket for the same incident.

9. If tickets are also inbound signals, map the service identity so the ticket joins the existing incident. An adopted customer-owned ticket keeps its status; **Write back** is explicit. ServiceNow correlation fields must follow the integration recipe to avoid an echo loop. **You should see** the adopted reference and source provenance.

   API: the source-specific Jira/ServiceNow/Azure DevOps webhook uses workspace webhook authentication. The platform marks the tickets it files — Jira labels `aiops` and `aiops-<incident id>`, ServiceNow `correlation_id` `aiops:<incident id>` with `correlation_display` `AIOps` — and drops its own tickets when they come back through the webhook, counted as `ticket_echo` in the response, so no vendor-side filter is needed; a ServiceNow inbound business rule must post `correlation_id` and `correlation_display` with the record, or the platform cannot tell its own ticket from a customer's while its create is still in flight. The payload fields and the business rule itself are in [Ticket systems → Inbound webhooks and echoes](/help/ticket-systems.md#inbound-webhooks-and-echoes).

## Verify and cleanup

Retain the source, platform notification/ticket audit and receiver receipts. Remove temporary targets and disable/remove only the temporary scoped route after the agreed test. Confirm the restored target/route state without exposing keys or URLs.

## Troubleshooting

| Symptom | Check | Next step |
|---|---|---|
| Pager reports no stored key | Target kind, region and key-presence result | Store the correct integration key through the private setup flow |
| Webhook received a message but no page exists | Actual target kind | It was a notification, not a page; configure a real pager to test paging |
| No ticket yet | The reason L3 gives: held, unmapped or no route | Resolve the named gate; do not assume connection existence enables filing |
| Write-back fails | Recorded external response and adopted-ticket ownership | Review the integration and its access; retry only the recorded operation |
| Approval seems missing from messages | Approval gate is not a push event | Poll Needs you or pending approvals |

## Related

[Alert sources](/help/setup-alert-sources.md) · [Ticket systems](/help/ticket-systems.md) · [Audit log → Notification setup checks](/help/audit-log.md#notification-setup-checks) · [REST API](/help/rest-api.md#what-rest-does-not-have).


## Next steps

- Next: [Ticket systems — Jira Cloud, ServiceNow, Azure DevOps](/help/ticket-systems.md)
- [All Connect your environment pages](/help/connect-your-environment/llms.txt)
- [Documentation home](/help/)