# Benchmarks and scores

> Golden datasets, the builtin suite, back-tests on your own incident and change history or an archived month, sandbox versus live access, the 80% gate, and what a score means.

_How-to · Updated 2026-10-26 · Describes release 0.1.164_

## Benchmarks: proving the agents are right {#benchmarks-proving-the-agents-are-right}

**Benchmarks** is where the pass rate on the executive report comes from, and it is the evidence to put in front of anyone deciding whether to trust the agents.

Open **Agent quality → Test Lab → Replay (recorded cases)**. The Test Lab destination appears with `benchmark.view` or `pillars.manage`; Replay needs `benchmark.view` to read and `benchmark.manage` to run, upload, import, override, re-run or resume. `/portal/benchmarks` remains a redirect to Replay.

The page opens on a **Latest score** hero — pass percentage and passed over total for the newest run — above the run history, which lists When, Run, Suite, Status, Pass rate, Agent access and Fingerprint.

### Golden datasets: the cases are yours

Requires `benchmark.view` to read, `benchmark.manage` to change. The **Golden datasets** card holds the case sets this tenant owns, separately from the runs that consumed them.

Uploading stores a set and **runs nothing**. Uploading and evaluating are separate acts: the same cases get graded many times — after a model change, a prompt edit, a new skill — and a set that only existed as a side effect of spending money could not be downloaded, re-run deliberately, or kept.

| Action | What it does |
| --- | --- |
| **Upload dataset** | Stores a `.csv` or `.json` golden set. Unusable rows are skipped, and the confirmation says how many and gives the first reason; a file with none usable is refused. |
| **Run** | Starts a graded run from that exact version. The dialog also picks the pillar and the access toggles. |
| **Download** | The cases back out, in the shape the uploader accepts — including their replay evidence. |
| **New version** | Appends v(N+1). The previous version is **kept**. Skipped rows are reported as on upload. |
| Trash icon | Moves the set to the trash. |

**A dataset is immutable.** Rename and redescribe freely, but the cases never change in place: a change is a new version. Runs cite the exact version they measured, so a score stays attached to the cases that produced it. If the cases could be edited underneath, `dataset_sha` would stop meaning "the same test" and every past number would become unfalsifiable.

Sets built by **Back-test on history** are stored as datasets too, marked `from history`, so they can be downloaded and re-run without re-deriving them from incidents — and a 34-case set with replay evidence is stored once rather than copied into every rerun.

### The trash: 30 days, then gone

Deleting a dataset moves it to the trash rather than destroying it. A benchmark set is expensive to assemble and easy to bin by accident.

- The **Trash** button switches the card to the deleted sets, each showing the date it will be destroyed.
- **Restore** brings one back.
- After the window (30 days on the shipped default), the scheduler destroys it permanently and records a `dataset.purge` audit event. The deadline is stamped on the row when you delete, so shortening the retention window later can never retroactively destroy something already binned.

Two things the trash does **not** touch. Runs already made from the set keep their scores — a score is the record of what the agent did on a given day, and binning the question does not un-ask it. And those runs stay re-runnable: each run keeps its own copy of the cases it measured, so **Re-run same scenarios** still works after the dataset is gone.

### Choosing the model

Every dialog that starts a run has a **Model** field. Leave it empty and the run uses tier routing, exactly as an alert-driven investigation would. Name a model and the run is pinned to it. The list of gateway model ids needs `pillars.manage`; without it the picker shows the hint *Listing gateway models needs pillars.manage. Choose Custom model id… and type the id.* and you type the id yourself.

Pinning sets **both** calls — the investigation loop and the structured wrap-up — and switches off mid-run tier escalation. Escalation exists to move a hard production incident onto a stronger model; inside a benchmark it would move the run onto a different model than the one being measured, which is the confound the whole exercise is meant to remove.

The chosen model is recorded on the run as a field, shown in the **Model** column, and printed in the exported report. It also feeds the model-tier suggestions. Older runs had their model inferred from a `model=…` string in the label, which meant one typo attributed a score to the wrong model; the recorded value now wins, and the label is only read for runs that predate it.

A re-run inherits the model of the run it replays, for the same reason it inherits the access mode: "we changed the prompt, re-test" must not silently change what is being measured as well.

### Agent access: sandbox by default

Every dialog that starts a run carries two toggles under **What the agent may reach**. Both ship off, and off is the setting that makes a benchmark mean anything.

| Toggle | Off (default) | On |
| --- | --- | --- |
| **Your live datasources** | *Sandbox*: the agent gets its persona, your knowledge base and the skill pack, and nothing else — no datasource tools, no MCP servers, no lakehouse, no `http_get`. It reasons from the alert and what it has been taught. | The full serving path, exactly as an alert-driven investigation runs. |
| **Web search and fetch** | No egress at all. | The agent may consult vendor documentation. Independent of the datasource toggle, and your platform's `AIOPS_WEB_TOOLS` switch still wins. |

Sandbox is the default because a benchmark is meant to measure the **agent**. An agent reading the live estate scores differently every hour: the same scenario passes in the morning and fails after a deploy, for reasons the agent had no part in. A sandbox run is repeatable, costs a fraction as much, hammers nothing in production, and answers the question a customer is actually asking — *how good is this agent and the knowledge we have given it?*

Turn the datasources on when the question is different: "does the whole serving path work end to end", usually once, as a pre-go-live check.

The two are **not comparable**, and the platform refuses to pretend otherwise. Each run records the mode it ran in and is badged `sandbox` or `live estate` in the history, the hero and the run detail. The automatic baseline for the gate only considers earlier runs of the same dataset **in the same mode**, and naming a cross-mode baseline explicitly returns a 409 rather than a smaller number.

A benchmark run also stays out of your operational record. It produces an investigation like any other run, but one marked as a test: it is absent from the investigation stream the Dashboard and the executive report read, and its spend is excluded from cost answers and model-tier suggestions. Deleting a run deletes those investigations with it.


### Run the builtin suite

Requires `benchmark.manage`. This proves the pipeline works before you spend anything on your own data.

1. Select **Run builtin suite**.
2. Choose a **Pillar**. The dropdown lists the seven pillar keys.
3. Confirm with **Run suite**. The confirmation warns that it uses model budget when a real gateway is configured. With no gateway model configured the platform runs its deterministic mock model, so the builtin pack proves the pipeline keyless.

You should see a new run appear in the history with the suite `builtin` and a `graded/total` counter that climbs to completion.

The builtin pack has 58 scenarios across five pillars: 52 for `incident`, three for `compliance`, and one each for `change`, `finops` and `secops`. There are **no builtin scenarios for `improvement` or `reporting`**: running the suite for either is refused with `no builtin scenarios ship for <pillar>; run a golden set instead`, and no run is created. Use a golden set for those pillars.

### Back-test on your own history — no upload needed

Requires `benchmark.manage`. The golden set you can produce fastest is the one you already have: the incidents your team closed, with the RCA a human read and accepted.

1. Select **Back-test on history**.
2. Pick the **Agent** the cases are for: **All incidents** (the default, every incident whatever its pillar), **Incident agent**, **Change agent** or **Compliance agent**. The run works the cases on that agent. The change agent is graded on changes, not incidents — see [Back-test the change agent on its own changes](#back-test-the-change-agent).
3. Set **Look back** (days), **Max cases**, and optionally a **Min human score**.
4. Set the access toggles — sandbox is the default and is what you want here.
5. Confirm with **Start back-test**.

Each qualifying incident becomes one scenario. The agent gets what it originally got — the correlated alert's labels and annotations — and is graded against what your people settled on: the incident's recorded root cause, the service the wrap-up named, and the fix that was proposed.

#### Evidence replay: why a sandbox back-test is still a fair test

The agent that solved an incident reached its answer through dozens of tool calls — Grafana annotations, log queries, `kubectl`, cloud APIs, the audit trail. Handing a sandboxed agent only the alert labels and asking for the same root cause is not a hard test; it is an impossible one. It grades the question, not the agent.

So a case built from history carries the **observations the original run recorded**. In the replayed run the agent gets a `read_evidence` tool: call it with no arguments for the index of what was collected, then with an index to read one in full. That is the only evidence available — no datasource is connected, and the estate is not reachable.

This is what makes the back-test both fair and durable. Fair, because the question becomes the one worth asking: *given this evidence, do you reach the same conclusion?* Durable, because it still has an answer a year later, when the pod that was crashlooping is long gone and the GPUs that were saturated are idle.

Evidence travels with the case through export and upload, so a golden set you download keeps its evidence and a set you upload replays with it. A case whose evidence is missing is a harder, different benchmark than the run it came from.

#### What is excluded, and why

The exporter copies what the platform recorded, and the platform recorded whatever the model produced on the day — including days the gateway was down. A case whose "right answer" is nonsense is worse than no case: it grades every future agent against nonsense and drags the headline number. So a case is dropped or repaired when:

| Recorded | What happens |
| --- | --- |
| The RCA is a model or gateway failure ("I'm having trouble reaching the model…") | Dropped. A run that never reached a model produced no analysis. |
| `root_cause_service` holds a paragraph instead of a name | Repaired: the alert's own `service` label is used instead. A 300-character sentence turns the service match into a token-overlap lottery. |
| The proposed fix is only the platform's own footer (`Execute via route '…'`) | The footer is stripped. If nothing remains, the case is dropped. |
| The root cause is a raw JSON blob | Salvaged with the engine's tolerant parser; if unsalvageable, the alert's service is used. |
| The title is a placeholder like `ERROR logs detected for [no value]` | The alert's own description annotation is folded into the case description. |

An incident qualifies when it is `resolved` or `escalated`, has a completed investigation, a recorded root cause **and** a proposed fix. Anything else is reported as skipped rather than quietly scored, and the response tells you how many were skipped and why. A window with nothing gradeable in it returns a 400 saying so, instead of an empty run that looks like a zero.

**Min human score** is the version of this an SRE lead asks for. Set it to 7 and only incidents whose investigation someone rated 7 or higher on the **Feedback** page are included — grade the agent against the reports you actually trusted, not against every RCA that was ever generated.

The run is labelled `history <days>d (<n> cases)` — `<agent> history <days>d (<n> cases)` when you picked an agent, for example `change history 90d (6 cases)` — carries the `golden_set` suite and is marked as coming from history, so it never gets confused with an uploaded file. The set is also stored as a golden dataset, so you can run it again without rebuilding it.

Compliance findings need no separate path: a Security Hub or Defender for Cloud finding opens a compliance-pillar incident, so **Compliance agent** back-tests those incidents by the rules above — a finding whose incident has a completed investigation, a recorded root cause and a proposed fix becomes a case. A compliance scan's control incident with no analysis run of its own is skipped as having no completed investigation.

### Back-test the change agent on its own changes {#back-test-the-change-agent}

Pick **Change agent** and the back-test replays your changes instead of your incidents: the agent drafts each plan again and is graded against what was executed and how it ended. It uses model budget per case, like any back-test.

A change counts only when its outcome is known:

- **Validated** — completed, and a validation reported the condition cleared.
- **Rolled back** or **failed** — with a recorded reason: the failed execution's detail, the verification note, or the rollback's detail.

Anything else — a change still planned or awaiting approval, a completed change no validation cleared, a failure with no reason on record — is reported as skipped rather than scored. So is a change whose origin incident was closed as noise.

The agent gets what the change planner got: the change's title, its target service and the request, the resources it was scoped to, and the observations its planning runs recorded (the same evidence replay as an incident case). The expected answer follows the outcome and the feedback on the planning runs:

| The change | Its feedback | The plan must |
| --- | --- | --- |
| Validated, executed as a pull request (`gitops_pr`) or a command (`cli`) | None, or rated up | Carry a rollback **and** use that same channel |
| Validated through another channel (`api`, `mcp`, `network`) | None, or rated up | Carry a rollback. A sandboxed agent has no connection to draft a typed step through, so the channel is not asserted |
| Rolled back or failed with a reason | None, or rated up | Carry a rollback |
| Any of the above | A human wrote a **correction** | Carry a rollback, and its fix must match the correction (the action rule in [What a score means](#what-a-score-means)) |
| Any | Rated **down** with no correction | **Excluded** — nobody said what the right plan was |

The plan is graded the way the Test Lab's change lane grades one: rollback and channel decide the verdict; blast radius and risk tier are shown beside it and never counted, because the platform fills those in, not the agent. The request text is the question, and a correct plan restates its request, so the words of the plan are not matched against it. A change-agent run asks the agent for the planner's wrap-up — a remediation plan — exactly as a production planning run does.

**Min human score** reads the scores on the planning runs. The API's `validated_only` keeps only validated changes. A window with no gradeable change returns a 400 that says a change needs a known outcome.

### Back-test from the archive {#back-test-from-the-archive}

A month that went past the retention window has left the database, so **Back-test on history** no longer sees it. If the record archive is on, an owner with `benchmark.manage` turns an archived month into a golden dataset from the **Archive** page — **Create back-test cases**, the same **Agent**, **Validated fixes only** and **Min human score**, and the same rules as above. The dataset shows here with a `from archive` badge, each case carries the archive batch id and the SHA-256 of its manifest, and nothing runs until you select **Run**. The steps are in [Archive → Turn a month into back-test cases](/help/archive.md#back-test-cases-from-the-archive).

### Export a golden set from your incidents

Requires `benchmark.view`. Same cases as the in-product back-test, written out as a file you own.

1. Select **Export golden set**.
2. Pick the **Agent**, then set **Look back**, **Max cases**, an optional **Min human score**, and **CSV** or **JSON**.
3. Select **Download**. Nothing runs and nothing is charged — this only reads what you already have. The confirmation tells you how many cases came out and how many incidents were skipped.

Ground truth is decided per incident, in this order:

| The incident's feedback | What becomes the expected answer |
| --- | --- |
| A human wrote a **correction** | The correction. Someone read the RCA, said it was wrong, and wrote down what should have happened — that is the best ground truth in the system, and it wins even over a thumbs-up. |
| Rated up, scored, or no feedback at all | The recorded root cause and the proposed fix. The incident closed and nobody objected, so the answer stands. |
| Rated **down** with no correction | **Excluded.** The agent was wrong and nobody said what right looks like; scoring against it would grade the agent on an answer no human ever endorsed. |

Each row carries a `verdict` column saying which of the first two applied (`corrected` or `accepted`) and the `incident_id` it came from, so a reviewer can trace any case back.

With **Change agent** picked, the file holds change cases instead, and every row also carries `pillar`, `change_id` (the change it came from) and `change_json` (what the plan is graded on, as a JSON object). Uploading that file reads them back, so a change set round-trips like an incident set; an export without change cases keeps exactly the columns above.

The file is the same shape **Upload golden set** accepts, on this deployment or any other. That is the point: the set outlives the deployment. Keep it in the repository beside the agent, diff the score across releases, hand it to a vendor doing an evaluation, or feed it to a harness that has never heard of this platform.

In the CSV, a cell that starts with `=`, `+`, `-`, `@`, a tab or a carriage return gets a leading apostrophe, so a spreadsheet keeps incident text as text instead of running it as a formula. **Upload golden set** removes that one apostrophe again; the JSON format is written unchanged.

Two habits worth forming. Export **before** a model or prompt change, so the dataset that produced your baseline is pinned as a file rather than as a database query whose answer drifts. And re-export periodically — every corrected RCA makes the set better, because a correction is a human telling you exactly what the agent should have said.

### Upload a golden set

Requires `benchmark.manage`. Use this when the cases live in another tracker rather than in this platform's own incident history.

1. Select **Upload golden set**.
2. Provide a `.csv` or `.json` file. Required columns are `description`, `root_cause` and `expected_action`; optional `id`, `service` and `labels` (or `labels_json`) improve grading. Give `service` where you can — it is the clean grading target, and without it the free-text `root_cause` is matched instead.
3. The upload runs in the background against the **incident** pillar.

```csv
id,service,description,root_cause,expected_action
INC-4471,checkout,"Checkout p99 latency 4s for 20 minutes","connection pool exhausted on checkout-db","restart the checkout pool / raise max connections"
INC-4488,ingress,"5xx spike on the public ingress","expired TLS certificate on the ingress listener","renew and reload the ingress certificate"
```

You should see a run row appear with the suite `golden_set`, your filename as its label, and `graded/total` climbing.

Check the count. A file whose header is missing a required column, or whose every row is unusable, is refused with a 400 that names the problem per row. If only *some* rows are bad they are skipped and the rest still run: the confirmation says how many rows were skipped and gives the first reason, and the API response lists the reasons in `skipped` (at most 50). Compare the run's scenario total against the number of rows in your file before you read anything into the score.

Or select **Import results** to record an external harness run — `aiopslab`, `itbench` or `sregym`. Pick the harness and pillar, paste the harness JSON, and use a label containing `model=<id>` so the run feeds the model-tier suggestions described in the **Agents and guardrails** chapter. An imported run carries results but no replayable scenarios, so **Re-run same scenarios** is refused on it and no gate verdict is computed for it.

### Prove the incident pillar clears 80% on your own data

This is the procedure to run before a go-live review. Requires `benchmark.manage`.

1. Export 30 to 100 resolved incidents from your existing tracker into the CSV above. Aim for a spread of services and failure modes, not only the dramatic ones.
2. Upload it with **Upload golden set** and wait for `graded/total` to reach the total.
3. Select the run to open it. Read the **Latest score** hero and the per-scenario table.
4. Work down the failures. Each row states which half failed — the root cause or the action. A root-cause failure usually means missing knowledge; write the runbook and activate it. An action failure requires checking the proposed operation, target evidence, opted-in connection and expected answer; adding a legacy route is not a prerequisite for prepared plans.
5. Where the automatic grade is simply wrong, select **Override** on the row, type why (the note is audited), then select the row's one button: **Mark pass** on a failing row, **Mark fail** on a passing one.
6. After changing a model, a prompt or a skill, open the same run and select **Re-run same scenarios**. This replays the exact scenario set as a fresh run.
7. Open the new run and read its gate block.

You should see it read `Gate: PASS`, with the run's pass rate against the same-dataset baseline and the floor (80% by default) beside it, and no regressions listed. Named regressions and improvements are listed by scenario id. With no earlier run over the same dataset there is no baseline, and the gate is the floor alone. The same holds across a grader upgrade: a case that flips between two grader versions moved with the grading rules, not the agent, so the automatic baseline only considers runs graded by the same version, and naming one graded by another returns a 409.

If a run is still marked `running` after a restart, open it and select **Resume (after a crash)**. Only the scenarios missing from its results re-run; resuming a finished run is refused with a conflict. A re-run inherits the source run's access mode, so "we changed the model, re-test" never silently changes what is being measured as well.

### Export a run, or throw it away

**Export report** renders the run as a markdown artifact: the headline number, what was measured (mode, dataset, agent fingerprint), the failures with the expectation beside them, and any adjudicated overrides. It is rendered by code, not by a model, so a customer-facing number has nothing in it to hallucinate. It lands on the **Artifacts** page, PII-scrubbed on write like every other artifact. Needs `benchmark.view`.

**Delete** removes the run, its per-scenario results and the sandbox investigations it produced. Needs `benchmark.manage`. Benchmark output is test output — unlike incidents and the audit trail it is meant to be disposable — so this is a real delete and it cannot be undone.

## What a score means {#what-a-score-means}

The grader evaluates the root-cause and action claims the case actually supplies:

- **Root cause:** for a service-name truth, the asserted `root_cause_service` decides **when the wrap-up carries that field**. An empty field is no answer; a prose mention cannot replace or outvote it. Wrap-up schemas without the field (other pillars) are read from their root-cause prose. A free-text truth is matched against the field plus prose: an affirmed mention, or at least half of its significant tokens (at least four characters, plus selected three-letter technical terms such as DNS, TLS and CPU). An asserted `root_cause_service` blaming a declared must-not-blame service fails the case.
- **Action:** match the case's action keys in the proposed fix or remediation plan's steps, commands and manual commands. Without action keys, match the expected action as a full string or at least half of its significant words; one stray word is not a match. Hyphens and underscores act as separators, and regular plurals count as their word. This is an action-class match, not an exact-command comparison.

A truth that states only a service or only an action is graded on that half; when it states both, both must match. A declared suspect-change truth adds its own check. Each case shows which check failed. Each case scored by this grader records its **grader version** (v2 from 0.1.109). Certificates name it when relevant result rows carry a version stamp; legacy unstamped results from this grader count as v1. Other graders may have no such row. Compare uses matching versions. The model report uses the newest grader present: v1 is omitted when v2 is present; v1-only results remain shown.

An operator override wins over the automatic grade and badges the row `SME override`. That is what makes an aggregate pass rate defensible in a review — a machine-graded number nobody was allowed to correct is not evidence.

Three identities matter when comparing results: the **fingerprint** identifies the agent’s model, prompt and skill pack; the **dataset hash** identifies the scenario set; the **grader version** identifies the matching rules. Keep all three with a score. A changed pass rate across a changed catalogue or grader does not isolate a change in agent quality.

## Use cases

### Deciding whether to trust the agents — engineering manager

Before switching an alert route to the incident agent, the manager exports 60 closed incidents, uploads them with **Upload golden set**, checks the run's scenario count matches the file, and reads the per-scenario failures. Two failures are grading mistakes and get an audited **Override**; four are genuine knowledge gaps and become runbooks. **Re-run same scenarios** after the runbooks are active shows the gate block flip to `Gate: PASS` with two named improvements and no regressions. That run, its fingerprint and its overrides are what goes in the change record.

## Troubleshooting

Symptoms specific to this page. Sign-in problems, missing nav entries in general, stuck or queued investigations, and `$0.0000` costs are in the **Troubleshooting** chapter.

| Symptom | Cause | What to do |
| --- | --- | --- |
| **Run suite** for `improvement` or `reporting` is refused with *no builtin scenarios ship for …* | No builtin scenarios ship for those pillars | Expected. Upload a golden set for those pillars instead. |
| **Re-run same scenarios** missing or refused | The run was imported from an external harness | Imported runs carry results, not replayable scenarios. Re-run the harness and import again. |
| Golden-set score looks wrong | Rows were skipped; the upload's confirmation said how many and why | Compare the run's scenario count with your file's row count, fix the offending rows, and upload again. |

## Next steps

- [Test Lab](/help/test-lab.md) — live fault injection, the lab box and how Live and Replay differ.
- [Set up the Test Lab](/help/test-lab-setup.md) — provision the lab before a live run.
- [Feedback](/help/feedback.md) — the human scores a back-test can filter on.
- [Artifacts](/help/artifacts.md) — where an exported report or a certification lands.
- [Knowledge base](/help/knowledge-base.md) — a root-cause failure usually means a missing runbook.
- [Knowledge, x-ray and insights](/help/intelligence.md) — the hub for every page of this group.