Operator runbook: the platform team

Platform monitoring and incidents, releases, backup, licence, security, patching, the weekly and monthly checks, support and the settings reference.

How-to · Updated 2026-11-02 · Describes release 0.1.184

9. Monitoring the platform itself

The platform cannot investigate its own outage. Watch it from your own monitoring system and page your own on-call. Never send platform alerts into the platform's own alert intake.

Full reference: Platform monitoring and alerting.

Check the platform is up

When to do it. At the start of each shift, after every release, and whenever a user reports the portal is slow or down.

Who. Anyone. The health endpoint needs no sign-in.

Steps.

1. Run:

   curl --fail --silent --show-error https://aiops.example.com/healthz
   

2. Read the answer. It is {"ok": true, "version": "<version>"}.

How to check it worked. ok is true and version is the release you expect (0.1.184 for this runbook).

If it did not. No answer or an error: go to Platform incidents. A different version: a deployment did not finish or rolled back; see Upgrade, verify and roll back.

Connect the metrics scrape

When to do it. Once, at installation. Again if the release name, namespace or metrics token changes.

Who. The customer monitoring or platform team, with access to the cluster and to your Prometheus.

Steps.

1. Metrics are served on a separate in-cluster port, 9464 (AIOPS_METRICS_PORT; 0 turns it off). It is never on the public application port.
2. The scrape needs the bearer token AIOPS_METRICS_TOKEN (chart value secrets.AIOPS_METRICS_TOKEN). The chart derives it from AIOPS_SECRET_KEY when left empty; with existingSecret, add the key yourself. Never run without it: without a token, any workspace connection aimed at port 9464 could read every workspace's metrics.
3. Point your Prometheus at the Services <release>-metrics (control plane) and <release>-runner-metrics (runners, usually namespace aiops-agents). With Prometheus Operator, set metrics.serviceMonitor.enabled.
4. Allow your Prometheus to reach the runners: set metrics.runnerScrapeFrom.namespaceSelector and podSelector. Empty selectors leave the runner port closed.
5. Import the Grafana dashboard deploy/grafana/aiops-platform-dashboard.json.

How to check it worked. From inside the cluster:

TOKEN=$(kubectl -n aiops get secret aiops-secrets \
  -o jsonpath='{.data.AIOPS_METRICS_TOKEN}' | base64 -d)
kubectl -n aiops run curl --rm -it --image=curlimages/curl -- \
  curl -s -H "Authorization: Bearer $TOKEN" http://aiops-metrics:9464/metrics | grep aiops_build_info

aiops_build_info shows the running version, and every expected target is UP in Prometheus. From outside the cluster, port 9464 must not answer.

If it did not. Targets down: check the selectors and the token. Runners answer even with empty selectors: your network plugin may not enforce NetworkPolicy. Chart settings alone do not prove isolation; test it.

Set up the platform alert rules

When to do it. Once, at installation, and after each release whose notes add a rule.

Who. The customer monitoring team.

Steps.

1. Copy the starter rules from Starter rules into your rule group. They are: AiopsRunnerSilent, AiopsRunnerNotReady, AiopsDbPoolExhausted, AiopsIngest5xx, AiopsIngestBusy, AiopsBudgetTrip, AiopsLeaderLost, AiopsAuditSealBacklog.
2. If the record archive is on (AIOPS_ARCHIVE_BUCKET), add AiopsArchiveStale.
3. Add AiopsChangeCaptureBehind (change capture more than one hour behind).
4. Route all of them in your own Alertmanager to the platform on-call, with resolved notifications on.
5. Confirm the RDS alarm topics have a confirmed subscriber. An alarm with no recipient pages no one.

Copy the expressions exactly. Two details matter: the pool rule must keep its "pool size above zero" clause, and the budget rule must keep its second clause, or the first trip is missed.

How to check it worked. All rules load as healthy. Fire one controlled test alert and keep the firing and resolved receipts from your on-call tool.

If it did not. A rule never fires in a test: check its labels and routing first. The thresholds are starting values, not service levels. Tune them to your environment.

What to watch day to day

Logs go to stdout. Forward them with your log pipeline.

10. Platform incidents

Use this section when the platform itself fails: the portal is down or slow, investigations do not start, or model calls fail. Estate incidents are in Handling one incident.

Full reference: Platform incident response.

Triage and contain

When to do it. A platform alert fires, /healthz fails, or users report the platform is not working.

Who. The platform on-call. Pausing agents needs a workspace owner (members.manage). Stopping one run needs incidents.act.

Steps.

1. Write down the UTC start time, the host name, the /healthz result and version, the workspace slug and any incident or investigation id.
2. Decide the scope: portal and API, alert intake, one runner, all runners, model calls, or one estate connection.
3. In the Ops console open Platform → Overview and read the Agent runners panel (see Runner not claiming).
4. If agent activity must stop for a workspace: Settings › Guardrails & cost, card AI budgets & kill switch, click Pause all agents. Alerts still arrive and incidents still open; new runs do not start and go to a person instead.
5. If only one investigation must stop: open the incident and click Stop this run. The stop takes effect at the next step; wait for the run to show Stopped.
6. Keep the evidence (deployment receipt, monitoring screenshots, error text) before any restart or rollback. Do not copy secrets, pod environment or Secret manifests.
7. Use the table below to find the owner and the fix.

How to check it worked. The alert resolves, /healthz shows ok: true, and queued work starts or is accounted for.

If it did not. Escalate as in Support and escalation.

Runner not claiming

When to do it. AiopsRunnerSilent or AiopsRunnerNotReady fires, or incidents sit with an analysis that never starts.

Who. A platform superadmin (Ops console) and the cluster team.

Steps.

1. In the Ops console open Platform → Overview. Scroll to Agent runners. The panel refreshes every 15 seconds.

2. Find the workspace row and read its badge:

   • runner live: a runner holds a fresh lease. The runner is working.
   • lease expired: a runner claimed work and stopped renewing it. The runner died or is stuck.
   • no claims: nothing is running. Normal when there is no work; a fault when work is queued.

3. Check the red N retries exhausted line. Each job gets 3 attempts (AIOPS_JOB_MAX_ATTEMPTS). A job that used them all has failed and needs a person to start a new run.

4. Check the runner pod:

   kubectl -n aiops-agents get pods
   kubectl -n aiops-agents describe pod <runner-pod>
   

5. Pod Pending: the cluster has no room. The cluster team frees capacity or raises the node pool limit. Raising an AI budget does not schedule a pod.

6. Pod running but the lease expired: read the pod's restart reason. A lease lasts 180 seconds (AIOPS_JOB_LEASE_S). After it runs out, eligible work resumes from its last saved step on the next runner.

7. Private-network agent: compare its version with the control plane. An older agent is not handed estate x-ray scans and may ignore newer settings. Roll it to the matching release.

How to check it worked. The row shows runner live while work runs, the queued investigations move to running, and the alert resolves.

If it did not. Do not delete database rows to clear a queue. Collect the evidence above and escalate.

Model gateway errors

When to do it. Investigations fail with a model, provider or authentication error, or the analysis reads mock, or cost stays at $0.00 while tokens climb.

Who. A platform superadmin and the model gateway owner.

Steps.

1. Open the failed investigation and copy the error text and the model id.
2. In the Ops console open Config → Platform defaults. Look at the Default model tiers pickers.
   • The pickers list your gateway's models: the endpoint and key work.
   • The page shows Gateway model list unavailable (…): the text in brackets is the failure. no gateway configured means AIOPS_API_BASE is empty.
3. Match the symptom:

4. After the cause is fixed, start a fresh analysis with Re-analyse on the incident. A re-run costs money; do it once.

How to check it worked. A new investigation names your model (not mock), shows non-zero tokens and, once the model is priced, a non-zero cost.

If it did not. Full table: Connecting the LLM gateway.

Other symptoms

Close a platform incident

Close only when: the release checks pass (Verify a release), queued work is accounted for, intake errors have cleared and audit sealing advances. A pod becoming Ready is not enough. Record the cause, actions, times and who owns the prevention work.

11. Upgrade, verify and roll back a release

The deployment pipeline is the normal way to upgrade. Choose the image and chart from a published release record, never from "the newest tag". This runbook describes release 0.1.184 (image tag 0.1.184, chart 0.1.184, git tag v0.1.184).

Full reference: Upgrade and release verification, Rollback, and the customer deploy runbook.

Upgrade to a new release

When to do it. A new release is approved for your installation. Book a maintenance window: the control plane is one replica and restarts with about a minute of downtime.

Who. The release team runs the pipeline. The customer approver approves the aiops-<slug> environment.

Steps.

1. Read the release notes of every release between the installed version and the target, and the post-upgrade checks table in Upgrade and release verification. Note any setting, key or IAM change.

2. Never install 0.1.106, 0.1.115 or 0.1.127. They are void.

3. Bring the customer branch up to the release:

   git checkout customer/<slug>
   git merge v0.1.184
   

4. Queue deploy-customer.yml from customer/<slug> with customer = the installation slug, imageTag = 0.1.184, chartVersion = 0.1.184, bootstrapDb = false.

5. Read every unit in the Plan stage. An application-only upgrade normally shows No changes. Stop and review anything unexpected before you approve.

6. Approve. The Deploy stage takes an RDS snapshot, waits for it, then runs helm upgrade --install --atomic --wait.

How to check it worked. Run Verify a release.

If it did not. If readiness fails, --atomic tries to roll back. Read the result; do not assume it worked. Then follow Roll back a release.

Verify a release

When to do it. After every upgrade and every rollback.

Who. Checks 2–6 need a platform superadmin. Check 7 needs a workspace member.

Steps.

For check 7, use the smoke script from a host that reaches the platform. Always pass the workspace's current webhook token. Without it the script mints a new token and every alert sender gets 401 until it is updated.

BASE=https://aiops.example.com TENANT=example-org \
  WEBHOOK_TOKEN="$(cat /secure/webhook-token.txt)" \
  ADMIN_PASSWORD_FILE=/secure/admin-password.txt scripts/customer-smoke.sh

How to check it worked. The script prints SMOKE OK and an incident URL. Keep the pipeline artifact deploy-log-<slug> (image tag, digest, chart, snapshot id) and the smoke result in the release record.

If it did not. The script makes one real model call and leaves its test incident open. HTTP 404 means the admin account is not a member of that workspace. A TENANT slug that does not exist creates a new workspace, and workspaces cannot be deleted: check the slug first.

Roll back a release

When to do it. A release that became ready misbehaves, and the previous image can still read the database.

Who. The release team, with customer approval. An emergency Helm rollback needs a principal listed in eks_admin_principal_arns.

Steps.

1. Check the key-rotation boundary first. Never roll back with Helm to a revision older than the last tenant secret-store key rotation. Stored connection secrets would become unreadable.

2. Normal path: queue deploy-customer.yml again with the previous imageTag. From 0.1.149 on, keep the current chartVersion (the newer chart gives the older image time to start). Leaving chartVersion empty uses the branch's chart.

3. Emergency path, inside the key boundary only:

   aws eks update-kubeconfig --name aiops-<slug>
   helm -n aiops history aiops
   helm -n aiops rollback aiops <revision> --wait
   

   Check the active kubectl context before running Helm.

4. Run Verify a release again.

How to check it worked. /healthz shows the previous version and all seven checks pass.

If it did not. If an image rollback is not enough, the data must be restored: see Backup, restore and DR. Some data a newer release changed (listed per release in Rollback) does not come back with an image rollback.

12. Backup, restore and disaster recovery

Plan for 5 to 10 minutes of possible data loss with point-in-time restore. The tested drill reached a verified database copy in about 22 minutes; switching the service over to that copy has not been drilled.

Full reference: Backup, restore and disaster recovery and Availability and recovery.

What is protected

Run the monthly restore check

When to do it. Monthly, in an approved window, and after any change to backup settings.

Who. An approved cloud operator in the source AWS account, with the release team.

Steps.

1. Follow Restore a side copy for verification. It restores a new instance next to the source. The source stays untouched.
2. On the copy, compare row counts at the restore point and replay every audit chain.
3. Record: restore point, lag, time until available, time until verified, count comparison, chain results, key-retention check.
4. Delete the copy and record the deletion.

How to check it worked. Counts match and every chain replays clean.

If it did not. A failed verification is a failed drill. Keep the cleaned evidence and investigate before you rely on the backups.

Recover the service

When to do it. The database is lost or corrupted, and a rollback of the image is not enough.

Who. Cloud team and release team together, with customer authorization.

Steps.

1. Restore a new instance (point-in-time, or the pre-upgrade snapshot). Keep the original.
2. Verify the copy as in the monthly check.
3. Follow Restore data when image rollback is insufficient: database login grant, controlled rename, control-plane restart without a deployment apply, then reconcile the Terraform state before the next deployment.
4. Restoring from before a key rotation: deploy the retained key as AIOPS_TENANT_SECRET_KEY_PREVIOUS.

How to check it worked. Health, login, a runner investigation, connection Test and a full audit-chain replay all pass. Only then release the old instance for deletion.

If it did not. Do not delete the original instance. Escalate as in Support and escalation. The cutover and state import are manual and not yet drilled; treat any timing as new evidence, not a promise.

13. Licence

The licence is an offline signed file. The control plane checks it at start-up and on every scheduler tick. It never calls a licence server.

Full reference: Licence.

Check the licence state

When to do it. Monthly, and whenever new runs are refused with a licence reason.

Who. A platform superadmin.

Steps.

1. In the Ops console open Config → Platform defaults. Find the Licence card.
2. Read the state badge and the Expires, Workspaces and Last verified lines.

While expired or invalid: alerts still arrive and incidents still open (and go to a person); sign-in, history, audit and exports keep working; rollback and ticket paths keep working.

How to check it worked. The badge reads valid and Expires is the date you expect.

If it did not. Renew (below). Do not change the system clock.

Install or renew the licence

When to do it. Before the expiry date in your operating calendar.

Who. The release or licence owner.

Steps.

1. Put the new token in AIOPS_LICENSE in the onboarding secrets file.

2. Update the pipeline variable group from the file:

   python3 scripts/customer-onboard.py ado <slug> --update-group
   

   This writes every value in the file and overwrites manual Library edits.

3. Queue the normal deploy-customer.yml run with the current image and chart. There is no upload page; renewal needs a rollout.

4. If the vendor changed the signing key, ship the new public key (licence.publicKey) and a token signed by it in the same rollout.

How to check it worked. Within a minute of start-up the Licence card shows valid with the new expiry, and the Ops Audit log has a licence.state_changed row.

If it did not. invalid after renewal usually means the key and token do not match. Keep licence renewal separate from secret-store key rotation.

14. Security operations

People, sign-in, the audit trail and staff access. Secret rotation and the record archive are in the full chapter.

Full reference: Security operations, Members and roles, Single sign-on, Audit log, Support access.

Add or remove a workspace member

When to do it. Someone joins, moves or leaves.

Who. A workspace owner (members.manage).

Steps.

1. Open Settings › Members & roles.
2. To add: click Invite member, enter Email, choose Role, click Invite member. Send the invite link through your normal channel.
3. To remove: on the member's row click Remove, then Remove member.
4. To change a role: pick the new role in the role selector on the member's row.

Builtin roles: owner, operator, approver, viewer, executive. Since 0.1.111 an operator cannot decide approvals (approvals.decide); an approver, executive or owner can.

How to check it worked. The member appears with the right role (or is gone), and the Audit log shows the change.

If it did not. A removed person may still hold API tokens or a single sign-on group. Check both.

Connect single sign-on

When to do it. Once per workspace, and when the identity provider's issuer, client secret or groups change.

Who. A workspace owner (members.manage).

Steps.

1. Create any custom roles you will map first.
2. Open Settings › Members & roles, card Single sign-on.
3. Fill Issuer URL, Client id, Client secret, Email domains (comma-separated) (for example example.com) and Default role for new logins (start with viewer).
4. Fill Group → role map (one per line: group = role), for example sre-team = operator.
5. Click Save identity provider, then Test discovery.

How to check it worked. Test discovery reports Discovery OK — …. A test user signs in through the provider and gets the mapped role.

If it did not. The error text is under the form. A group can never be mapped to a role that holds members.manage; that is by design. Details: Single sign-on.

Manage platform staff accounts

When to do it. A platform staff member joins or leaves, or forgets a password.

Who. A platform superadmin.

Steps.

1. In the Ops console open Platform → Users.
2. New account: Create user.
3. Leaver: click Disable on the row. The account's sessions and tokens are refused from the next request.
4. Forgotten password: Reset password or Email reset link.

How to check it worked. The row's state is correct, and the Ops Audit log records the action.

If it did not. The bootstrap admin (admin@local) is re-enabled at every start-up from the deployment settings, by design. Rotate its password in the vault file as well as in the console.

Verify the audit chain

When to do it. Monthly, after any database restore, and whenever tampering is suspected.

Who. A workspace member with report.view for a workspace chain. A platform superadmin for the platform chain.

Steps.

1. Workspace: open Settings › Audit log and click Verify chain. Tick Full replay first after a restore or a suspicion.

2. Platform: in the Ops console open Config → Audit log and use Verify chain (or Verify chains).

3. For an auditor or the monthly archive: click Export audit, choose From, To and Format NDJSON, then Download. Use Download the next file until no more rows remain.

4. Check each file offline:

   python3 scripts/verify-audit-export.py audit-export.ndjson
   

How to check it worked. The page reads Chain intact (or Chain verified · newest N rows). The offline check prints OK with no gap: line. Store the files in write-once storage.

If it did not. N newer rows await sealing is normal for about a minute. Chain broken at seq N means a sealed row was changed outside the application: export the trail as evidence and escalate as a security issue. Do not delete or edit audit rows.

Give or end staff support access

When to do it. Platform staff need to see a workspace's data to help with a problem.

Who. A platform superadmin grants. Any workspace member with members.manage can end it.

Steps.

1. Staff: in the Ops console open the workspace's detail page. In Staff data access click Grant myself support access (24h, audited), then Grant support access.
2. Staff: when done, click Revoke my support access, then Revoke access.
3. Customer: to end it early, open Settings › Members & roles and click Remove on the row with the support badge.

The grant is an operator membership. Like any operator it cannot approve or release a change (a standard change excepted), or change members, roles or single sign-on. It expires after 24 hours (AIOPS_SUPPORT_ACCESS_TTL_S).

How to check it worked. While active: a SUPPORT ACTIVE badge in the Ops console and a support badge in Members. The audit log has support.granted and, at the end, support.revoked.

If it did not. already a member — support grant not needed: the staff member already holds a real role there. If staff self-grant is turned off on the deployment, the workspace owner must invite the staff member instead.

Monthly access review

When to do it. Monthly. Keep a signed record.

Who. The workspace owner and the platform superadmin.

Steps.

1. Ops console Platform → Users: active staff, superadmins and unused accounts.
2. Each workspace Members & roles: members, roles and pending invites.
3. Single sign-on domains and group mappings against your joiner, mover and leaver list.
4. Open support grants. Revoke those no longer needed.
5. API tokens (Settings › Alert intake & tokens) and agent tokens: owners and expiry.
6. Use Audit log filters as evidence. Record the reviewer, the period and any exceptions.

How to check it worked. Every account and grant has a named owner and a reason. There is no built-in access-review report; the record is yours.

If it did not. Revoke what has no owner, then record it.

Rotate secrets

Secrets are rotated through the onboarding secrets file and a pipeline deployment. The tenant secret-store key needs a two-key procedure with a census before and after: stop if any row is unreadable. Follow Secret inventory and rotation step by step. Never Helm-rollback across a key rotation.

15. Patching and cluster upkeep

The cloud team owns the cluster version, capacity and certificates. The release team owns the application images and their dependencies.

Full reference: Patching and cluster upkeep.

Monthly image and dependency review

When to do it. Monthly.

Who. The release team, with the registry operator.

Steps.

1. Find the running image tag: version from /healthz and the tag in deploy-log.json. Review that image, not the newest build.
2. Read the registry's enhanced scan findings for that tag (full, paginated findings or Inspector results; the summary count is not complete).
3. Record severity, package, fixed version, scan time, image digest and how you counted.
4. A fix needed: the release team updates the pinned dependency, builds a new version, scans it, and ships it through Upgrade a release.

How to check it worked. The record is filed and compared with last month using the same counting method.

If it did not. A scanner is detection only. It does not patch or accept anything.

Kubernetes minor-version upgrade

When to do it. Before the cluster's Kubernetes version leaves AWS support. One minor version at a time.

Who. The cloud team, with the release team.

Steps.

1. Check the installed version, the support horizon and the EKS upgrade insights. Review deprecated APIs, add-ons and your own controllers.
2. Change cluster_version in infrastructure/iac/envs/<slug>/env.hcl on the customer branch.
3. Plan and apply through the approved infrastructure workflow. Read every change before approving.
4. Verify nodes, application health, volume attachment, metrics, runner claims, sign-in and audit sealing. Then run Verify a release.

How to check it worked. All checks pass. Record the actual downtime.

If it did not. This procedure has not been drilled on a customer cluster. Stop, keep the evidence and escalate.

Nodes, capacity and certificates

When to do it. Monthly.

Who. The cloud team.

Steps.

1. Nodes: EKS Auto Mode replaces nodes about weekly and at 14-day expiry. Check Ready nodes, pending pods and requests against capacity and quota. The control plane can only move to a node in the zone of its /data volume.
2. Database: RDS connections, free storage and the storage autoscaling ceiling. Confirm the alarm topics have a confirmed recipient.
3. Certificates: the load balancer uses ACM. DNS-validated certificates renew themselves only while their validation records still resolve: keep those records. Imported certificates need manual renewal.
4. Put every expiry in one calendar: certificates, API tokens, agent tokens, the model gateway key and the licence.

How to check it worked. No pending pods, no alarm without a recipient, no expiry inside the next month without an owner.

If it did not. Capacity faults: Runner not claiming.

16. Weekly and monthly checks

The daily routine (section 2) handles what needs a person today. These checks look at trends: is the service getting better, is the noise going down, and is the AI spend where it was agreed.

Assign a named owner to each check. Keep what you found and what you decided in the monthly record (16.5). A tick without a figure is not evidence.

Permissions: Report and Usage & billing need report.view. Agents activity needs incidents.view; its cost figures also need report.view. All built-in roles hold report.view and incidents.view.

16.1 Weekly: agent spend and failures

When to do it. Once a week, and after any change to models, guardrails or connections.

Who. A workspace owner, or anyone with report.view who reports to them.

Steps.

1. Open Insights › Agents activity.
2. Select Last 7 days.
3. Read the tiles: Running now, Queued, Waiting for a person, Runs, Failed and Cost (with the cost per run under it).
4. Look at each pillar card. Compare runs, failures and cost with last week. Select Only these on a card to filter the list below to that pillar.
5. In the list below, select Runs. Look at the failed runs and at the most expensive ones. Each row links to its incident, finding or change. The column for why a run started uses plain words, for example An alert arrived, A person asked, Checking a fix held and, from 0.1.184, Re-analysed after a fix was rolled back.
6. Select Audit trail to see what the agents did, in the audit log's words.
7. Read the Test Lab card separately. Test spend is never part of the figures above.

How to check it worked. You can state, for the week: total runs, failed runs, total cost and the costliest pillar, and you have looked at every failed run.

What to do with what you find.

• Cost per run rises with no rise in incidents: check whether the model tier or the cost cap changed (Settings › AI models, Settings › Guardrails & cost).
• Many runs with An alert arrived on findings: check the scanner sync and the burst limit (sections 8.5 and 7.4).
• Many failed runs: open a few. Model gateway errors belong to the platform team (section 10). Runs blocked by a limit or by Pause all agents show as failed too, with the reason.
• Waiting for a person above 0 for days: someone owes a decision. Take it to Home › Needs you.
• Cost shows $0.00 while runs happen: the model is not priced (section 7.2). Do not report that figure.

If it did not work. The page says it needs a permission: ask a workspace owner to grant incidents.view, or report.view for the cost columns. Queued reads private-network agent only when the workspace runs its agents in the platform; the queue count applies to a private-network agent.

16.2 Monthly: noise per month

The platform counts, for each UTC calendar month, how many monitoring alerts arrived and what they became. This is the clearest figure for "how much noise did the platform take off the team".

The columns are: Alerts received (repeats included), Duplicates dropped, Joined an incident, Incidents, Grouped (incidents merged into a related one), Closed as noise and Noise reduction. Scanner findings are counted in their own table, Scanner findings · per month. Test alerts are not counted.

When to do it. At the start of each month, for the month that just ended. A glance each week is useful while you tune intake rules.

Who. The operations lead, or anyone with incidents.view.

Steps.

1. Open Home › Dashboard.
2. Near the bottom, open the fold Alerts → incidents · per month. It shows the last six UTC months.
3. Write down, for last month: alerts received, incidents, closed as noise and the noise reduction.
4. Compare with the month before.
5. For the same figures beside the rest of the service report, open Insights › Report, section 2 · Noise & correlation.

Through the API: GET /api/aiops/{tenant_id}/dashboard/noise (MCP tool dashboard_noise).

How to check it worked. The two places show the same month figures. Both are counted by the server from what intake recorded when each alert arrived, so they have no page limit.

What to do with what you find.

• Closed as noise is high for one source or service: that source alerts on things nobody acts on. Fix it at the source, or add an intake rule (Settings › Alert intake & tokens, fold Intake rules, service aliases and test alert).
• Grouped is low while storms happen: check the dependency map (Incidents › Service dependencies). Grouping by relationship needs declared or traced dependencies.
• Duplicates dropped is very high: a source re-sends the same alert. Check its repeat interval.

If it did not work. The fold is missing: the workspace has no recorded month yet. A new workspace shows it after its first alerts.

16.3 Monthly: the service-delivery report

When to do it. Once a month, before the service review with the customer. Use 30d, or 90d for a quarterly review.

Who. The operations lead. Reading the report needs report.view; some sections also need incidents.view or benchmark.view. A section your role cannot read says which permission it needs, and the rest of the report still shows.

Steps.

1. Open Insights › Report.
2. Pick the period at the top: 7d, 30d or 90d. It applies to every section.
3. Read the four tiles at the top: Incidents opened, Incidents resolved, Platform changes and Scanner-cleared score. A Watch: line under the title names what needs attention.
4. Read the sections in order:
   1. Service levels — opened, resolved, within target, open backlog, the table by priority, time to first analysis and time to recovery.
   2. Noise & correlation — the monthly noise tables (16.2).
   3. Remediation outcomes — changes proposed or made: automatic against human-approved, validated, failed, rolled back, by risk tier, and the approval turnaround.
   4. Compliance posture — the scanner-cleared score now against the start of the period, unresolved findings by severity, suppressed findings and fixes awaiting the scanner.
   5. Recurrence & problems — open problems, automation candidates, problems blocked because a fix did not hold.
   6. Human feedback & agent quality — analysed automatically, root cause written, Test Lab pass rate and reviewer score.
5. Open Appendix · Platform cost (FinOps) only if cost is on the agenda. It is closed by default on purpose: AI spend is a platform cost, not a service outcome.
6. Select Download report to save the same report as a Markdown file. It is a local download; it does not create an artifact.

How to check it worked. You have the downloaded file for the period, and each section shows figures, not a permission notice.

What to do with what you find.

• Within target drops: look at the per-priority table and the backlog by age. Old open incidents usually wait on a decision in Needs you.
• Failed or Rolled back changes rise: read those changes (section 5) and check whether a runbook was demoted (section 6).
• An automation candidate appears under Recurrence & problems: the same fix keeps working. Consider certifying a runbook for it (section 6).
• A figure shows + after it: the period holds more than one page of rows (500). Use a shorter period, or count from the list pages.

Targets. A resolved incident is within target when its condition cleared inside the target for its priority: P1 4 hours, P2 8, P3 24, P4 72, unless your workspace sets sla_targets_h. Compliance findings and incidents closed as noise are not counted as incidents.

If it did not work. The Test Lab pass rate tile reads needs benchmark.view: the reader lacks benchmark.view. It reads no finished Test Lab run yet until a run has finished. More in Service delivery report.

16.4 Monthly: the spend statement

When to do it. After the 1st of each month, for the month that just ended.

Who. A workspace owner, with finance. Needs report.view.

Steps.

1. Open Insights › Usage & billing and pick the month with the month buttons.
2. Read Budget · this month against your Monthly limit (USD) (section 7.2), then Where the spend goes, Daily spend, Costliest incidents and the fold Cost by model (on 0.1.183 a section named By model).
3. Select Download CSV and file it with the monthly record.
4. Open Insights › Agents activity, select Last 30 days, and check that the cost by pillar matches what you expect. Both pages read the same run records.
5. Compare the platform's figure with the model provider's invoice. Any model the price card does not price shows as an estimate or $0.00; note it.

How to check it worked. The CSV total matches the page. The difference to the provider invoice is explained (unpriced models, test spend, other workspaces on the same provider account).

If it did not work. A month with no spend shows no rows. If spend looks too low, check model pricing first (section 7.2). Reconciliation and export are in Cost management.

16.5 The monthly record

Keep one short record per month, outside the platform where your process requires it.

The full weekly and monthly checklist for the platform team (backups, access review, patching, licence) is Recurring checklist.

17. Support and escalation

Support contacts and response times are not agreed yet. This section says what to send and in what order to escalate. It is not a service-level agreement.

Full reference: Support and escalation.

Contacts

Choose a severity

These levels are for support requests only. They are not an alert's severity, an incident's P1–P4 or a change's L0–L3.

Report a problem

When to do it. You cannot restore service yourself, or you suspect a product defect.

Who. The workspace operator or the platform on-call.

Steps. Send, through the agreed channel:

1. A short title, the severity and the business impact.

2. When it started (with time zone), the workspace, and whether it affects everyone or one role.

3. Steps to reproduce, what you expected and what happened. Any workaround, and any recent deployment, connection or setting change.

4. The incident id and investigation id (and change or scan id when relevant). Ids help more than screenshots.

5. The live build:

   curl --fail --silent --show-error https://aiops.example.com/healthz
   

6. An audit export for the affected period: Settings › Audit log → Export audit (report.view), NDJSON format. Platform staff use the Ops Config → Audit log.

7. Error text and, if useful, a screenshot with secrets and unrelated personal data removed.

Never attach passwords, API keys, bearer tokens, private keys or a kubeconfig.

How to check it worked. Support confirms receipt and the severity.

If it did not. Escalate one step (below).

Escalation path

1. Workspace operator. Find the scope, collect the evidence above, record what you already did.
2. Platform operator. Check the build, runner health, model errors and the audit trail. Tell a provider outage apart from a platform fault. To read a workspace's data, use a support access grant.
3. Engineering support. Gets the report with ids, build and evidence.
4. Service owner or security contact. For lasting business impact, suspected compromise, or a recovery decision that needs sign-off.

Contact: to be agreed between your organisation and the vendor (named owner for each step).

Close the support record with the confirmed cause, the action taken, the verification result and any follow-up.

Appendix A. Settings reference and defaults

This appendix lists the settings an operator most often needs, with their shipped defaults in release 0.1.184. The complete list, with every key, shape, permission and card, is Settings reference.

How settings are layered

1. Platform default. Set by a platform administrator with an environment variable (AIOPS_…) or, for a few, in the Ops console. Applies to every workspace that has not set its own value.
2. Workspace override. The workspace's own value. For on/off switches, true or false overrides and null returns to the platform default.
3. Agent override. A few fields (model, schedule, prompt, extra approval gates) live on each agent in Settings › Agent behaviour.

Workspace settings on the Guardrails page need members.manage. Every save is written to the audit log.

Safety and spend (Settings › Guardrails & cost)

Behaviour switches that ship on

Each can be turned off per workspace. All default on: aiops_unmapped_triage, aiops_rollback_first, aiops_runbook_standard_change, aiops_runbook_owns_operations, aiops_pir_at_submission, aiops_reopen_raise_severity, aiops_change_principal_check, aiops_change_recheck_mapped, aiops_backup_before_change, aiops_triage_checklist, aiops_reanalyse_after_rollback, aiops_storm_agent.

Default off: aiops_unauthorized_change_incidents, aiops_runbook_multi_cover.

What each one does: Settings reference.

Platform settings (environment, platform team only)

Platform settings are not audited inside the product. Record changes to them in your own change management.

Next steps

• Next: Upgrade and release verification
• All Install and operate the platform pages
• Documentation home