Benchmarks and scores

Golden datasets, the builtin suite, back-tests on your own incident and change history or an archived month, sandbox versus live access, the 80% gate, and what a score means.

How-to · Updated 2026-10-26 · Describes release 0.1.164

Benchmarks: proving the agents are right

Benchmarks is where the pass rate on the executive report comes from, and it is the evidence to put in front of anyone deciding whether to trust the agents.

Open Agent quality → Test Lab → Replay (recorded cases). The Test Lab destination appears with benchmark.view or pillars.manage; Replay needs benchmark.view to read and benchmark.manage to run, upload, import, override, re-run or resume. /portal/benchmarks remains a redirect to Replay.

The page opens on a Latest score hero — pass percentage and passed over total for the newest run — above the run history, which lists When, Run, Suite, Status, Pass rate, Agent access and Fingerprint.

Golden datasets: the cases are yours

Requires benchmark.view to read, benchmark.manage to change. The Golden datasets card holds the case sets this tenant owns, separately from the runs that consumed them.

Uploading stores a set and runs nothing. Uploading and evaluating are separate acts: the same cases get graded many times — after a model change, a prompt edit, a new skill — and a set that only existed as a side effect of spending money could not be downloaded, re-run deliberately, or kept.

A dataset is immutable. Rename and redescribe freely, but the cases never change in place: a change is a new version. Runs cite the exact version they measured, so a score stays attached to the cases that produced it. If the cases could be edited underneath, dataset_sha would stop meaning "the same test" and every past number would become unfalsifiable.

Sets built by Back-test on history are stored as datasets too, marked from history, so they can be downloaded and re-run without re-deriving them from incidents — and a 34-case set with replay evidence is stored once rather than copied into every rerun.

The trash: 30 days, then gone

Deleting a dataset moves it to the trash rather than destroying it. A benchmark set is expensive to assemble and easy to bin by accident.

• The Trash button switches the card to the deleted sets, each showing the date it will be destroyed.
• Restore brings one back.
• After the window (30 days on the shipped default), the scheduler destroys it permanently and records a dataset.purge audit event. The deadline is stamped on the row when you delete, so shortening the retention window later can never retroactively destroy something already binned.

Two things the trash does not touch. Runs already made from the set keep their scores — a score is the record of what the agent did on a given day, and binning the question does not un-ask it. And those runs stay re-runnable: each run keeps its own copy of the cases it measured, so Re-run same scenarios still works after the dataset is gone.

Choosing the model

Every dialog that starts a run has a Model field. Leave it empty and the run uses tier routing, exactly as an alert-driven investigation would. Name a model and the run is pinned to it. The list of gateway model ids needs pillars.manage; without it the picker shows the hint Listing gateway models needs pillars.manage. Choose Custom model id… and type the id. and you type the id yourself.

Pinning sets both calls — the investigation loop and the structured wrap-up — and switches off mid-run tier escalation. Escalation exists to move a hard production incident onto a stronger model; inside a benchmark it would move the run onto a different model than the one being measured, which is the confound the whole exercise is meant to remove.

The chosen model is recorded on the run as a field, shown in the Model column, and printed in the exported report. It also feeds the model-tier suggestions. Older runs had their model inferred from a model=… string in the label, which meant one typo attributed a score to the wrong model; the recorded value now wins, and the label is only read for runs that predate it.

A re-run inherits the model of the run it replays, for the same reason it inherits the access mode: "we changed the prompt, re-test" must not silently change what is being measured as well.

Agent access: sandbox by default

Every dialog that starts a run carries two toggles under What the agent may reach. Both ship off, and off is the setting that makes a benchmark mean anything.

Sandbox is the default because a benchmark is meant to measure the agent. An agent reading the live estate scores differently every hour: the same scenario passes in the morning and fails after a deploy, for reasons the agent had no part in. A sandbox run is repeatable, costs a fraction as much, hammers nothing in production, and answers the question a customer is actually asking — how good is this agent and the knowledge we have given it?

Turn the datasources on when the question is different: "does the whole serving path work end to end", usually once, as a pre-go-live check.

The two are not comparable, and the platform refuses to pretend otherwise. Each run records the mode it ran in and is badged sandbox or live estate in the history, the hero and the run detail. The automatic baseline for the gate only considers earlier runs of the same dataset in the same mode, and naming a cross-mode baseline explicitly returns a 409 rather than a smaller number.

A benchmark run also stays out of your operational record. It produces an investigation like any other run, but one marked as a test: it is absent from the investigation stream the Dashboard and the executive report read, and its spend is excluded from cost answers and model-tier suggestions. Deleting a run deletes those investigations with it.

Run the builtin suite

Requires benchmark.manage. This proves the pipeline works before you spend anything on your own data.

1. Select Run builtin suite.
2. Choose a Pillar. The dropdown lists the seven pillar keys.
3. Confirm with Run suite. The confirmation warns that it uses model budget when a real gateway is configured. With no gateway model configured the platform runs its deterministic mock model, so the builtin pack proves the pipeline keyless.

You should see a new run appear in the history with the suite builtin and a graded/total counter that climbs to completion.

The builtin pack has 58 scenarios across five pillars: 52 for incident, three for compliance, and one each for change, finops and secops. There are no builtin scenarios for improvement or reporting: running the suite for either is refused with no builtin scenarios ship for <pillar>; run a golden set instead, and no run is created. Use a golden set for those pillars.

Back-test on your own history — no upload needed

Requires benchmark.manage. The golden set you can produce fastest is the one you already have: the incidents your team closed, with the RCA a human read and accepted.

1. Select Back-test on history.
2. Pick the Agent the cases are for: All incidents (the default, every incident whatever its pillar), Incident agent, Change agent or Compliance agent. The run works the cases on that agent. The change agent is graded on changes, not incidents — see Back-test the change agent on its own changes.
3. Set Look back (days), Max cases, and optionally a Min human score.
4. Set the access toggles — sandbox is the default and is what you want here.
5. Confirm with Start back-test.

Each qualifying incident becomes one scenario. The agent gets what it originally got — the correlated alert's labels and annotations — and is graded against what your people settled on: the incident's recorded root cause, the service the wrap-up named, and the fix that was proposed.

Evidence replay: why a sandbox back-test is still a fair test

The agent that solved an incident reached its answer through dozens of tool calls — Grafana annotations, log queries, kubectl, cloud APIs, the audit trail. Handing a sandboxed agent only the alert labels and asking for the same root cause is not a hard test; it is an impossible one. It grades the question, not the agent.

So a case built from history carries the observations the original run recorded. In the replayed run the agent gets a read_evidence tool: call it with no arguments for the index of what was collected, then with an index to read one in full. That is the only evidence available — no datasource is connected, and the estate is not reachable.

This is what makes the back-test both fair and durable. Fair, because the question becomes the one worth asking: given this evidence, do you reach the same conclusion? Durable, because it still has an answer a year later, when the pod that was crashlooping is long gone and the GPUs that were saturated are idle.

Evidence travels with the case through export and upload, so a golden set you download keeps its evidence and a set you upload replays with it. A case whose evidence is missing is a harder, different benchmark than the run it came from.

What is excluded, and why

The exporter copies what the platform recorded, and the platform recorded whatever the model produced on the day — including days the gateway was down. A case whose "right answer" is nonsense is worse than no case: it grades every future agent against nonsense and drags the headline number. So a case is dropped or repaired when:

An incident qualifies when it is resolved or escalated, has a completed investigation, a recorded root cause and a proposed fix. Anything else is reported as skipped rather than quietly scored, and the response tells you how many were skipped and why. A window with nothing gradeable in it returns a 400 saying so, instead of an empty run that looks like a zero.

Min human score is the version of this an SRE lead asks for. Set it to 7 and only incidents whose investigation someone rated 7 or higher on the Feedback page are included — grade the agent against the reports you actually trusted, not against every RCA that was ever generated.

The run is labelled history <days>d (<n> cases) — <agent> history <days>d (<n> cases) when you picked an agent, for example change history 90d (6 cases) — carries the golden_set suite and is marked as coming from history, so it never gets confused with an uploaded file. The set is also stored as a golden dataset, so you can run it again without rebuilding it.

Compliance findings need no separate path: a Security Hub or Defender for Cloud finding opens a compliance-pillar incident, so Compliance agent back-tests those incidents by the rules above — a finding whose incident has a completed investigation, a recorded root cause and a proposed fix becomes a case. A compliance scan's control incident with no analysis run of its own is skipped as having no completed investigation.

Back-test the change agent on its own changes

Pick Change agent and the back-test replays your changes instead of your incidents: the agent drafts each plan again and is graded against what was executed and how it ended. It uses model budget per case, like any back-test.

A change counts only when its outcome is known:

• Validated — completed, and a validation reported the condition cleared.
• Rolled back or failed — with a recorded reason: the failed execution's detail, the verification note, or the rollback's detail.

Anything else — a change still planned or awaiting approval, a completed change no validation cleared, a failure with no reason on record — is reported as skipped rather than scored. So is a change whose origin incident was closed as noise.

The agent gets what the change planner got: the change's title, its target service and the request, the resources it was scoped to, and the observations its planning runs recorded (the same evidence replay as an incident case). The expected answer follows the outcome and the feedback on the planning runs:

The plan is graded the way the Test Lab's change lane grades one: rollback and channel decide the verdict; blast radius and risk tier are shown beside it and never counted, because the platform fills those in, not the agent. The request text is the question, and a correct plan restates its request, so the words of the plan are not matched against it. A change-agent run asks the agent for the planner's wrap-up — a remediation plan — exactly as a production planning run does.

Min human score reads the scores on the planning runs. The API's validated_only keeps only validated changes. A window with no gradeable change returns a 400 that says a change needs a known outcome.

Back-test from the archive

A month that went past the retention window has left the database, so Back-test on history no longer sees it. If the record archive is on, an owner with benchmark.manage turns an archived month into a golden dataset from the Archive page — Create back-test cases, the same Agent, Validated fixes only and Min human score, and the same rules as above. The dataset shows here with a from archive badge, each case carries the archive batch id and the SHA-256 of its manifest, and nothing runs until you select Run. The steps are in Archive → Turn a month into back-test cases.

Export a golden set from your incidents

Requires benchmark.view. Same cases as the in-product back-test, written out as a file you own.

1. Select Export golden set.
2. Pick the Agent, then set Look back, Max cases, an optional Min human score, and CSV or JSON.
3. Select Download. Nothing runs and nothing is charged — this only reads what you already have. The confirmation tells you how many cases came out and how many incidents were skipped.

Ground truth is decided per incident, in this order:

Each row carries a verdict column saying which of the first two applied (corrected or accepted) and the incident_id it came from, so a reviewer can trace any case back.

With Change agent picked, the file holds change cases instead, and every row also carries pillar, change_id (the change it came from) and change_json (what the plan is graded on, as a JSON object). Uploading that file reads them back, so a change set round-trips like an incident set; an export without change cases keeps exactly the columns above.

The file is the same shape Upload golden set accepts, on this deployment or any other. That is the point: the set outlives the deployment. Keep it in the repository beside the agent, diff the score across releases, hand it to a vendor doing an evaluation, or feed it to a harness that has never heard of this platform.

In the CSV, a cell that starts with =, +, -, @, a tab or a carriage return gets a leading apostrophe, so a spreadsheet keeps incident text as text instead of running it as a formula. Upload golden set removes that one apostrophe again; the JSON format is written unchanged.

Two habits worth forming. Export before a model or prompt change, so the dataset that produced your baseline is pinned as a file rather than as a database query whose answer drifts. And re-export periodically — every corrected RCA makes the set better, because a correction is a human telling you exactly what the agent should have said.

Upload a golden set

Requires benchmark.manage. Use this when the cases live in another tracker rather than in this platform's own incident history.

1. Select Upload golden set.
2. Provide a .csv or .json file. Required columns are description, root_cause and expected_action; optional id, service and labels (or labels_json) improve grading. Give service where you can — it is the clean grading target, and without it the free-text root_cause is matched instead.
3. The upload runs in the background against the incident pillar.

id,service,description,root_cause,expected_action
INC-4471,checkout,"Checkout p99 latency 4s for 20 minutes","connection pool exhausted on checkout-db","restart the checkout pool / raise max connections"
INC-4488,ingress,"5xx spike on the public ingress","expired TLS certificate on the ingress listener","renew and reload the ingress certificate"

You should see a run row appear with the suite golden_set, your filename as its label, and graded/total climbing.

Check the count. A file whose header is missing a required column, or whose every row is unusable, is refused with a 400 that names the problem per row. If only some rows are bad they are skipped and the rest still run: the confirmation says how many rows were skipped and gives the first reason, and the API response lists the reasons in skipped (at most 50). Compare the run's scenario total against the number of rows in your file before you read anything into the score.

Or select Import results to record an external harness run — aiopslab, itbench or sregym. Pick the harness and pillar, paste the harness JSON, and use a label containing model=<id> so the run feeds the model-tier suggestions described in the Agents and guardrails chapter. An imported run carries results but no replayable scenarios, so Re-run same scenarios is refused on it and no gate verdict is computed for it.

Prove the incident pillar clears 80% on your own data

This is the procedure to run before a go-live review. Requires benchmark.manage.

1. Export 30 to 100 resolved incidents from your existing tracker into the CSV above. Aim for a spread of services and failure modes, not only the dramatic ones.
2. Upload it with Upload golden set and wait for graded/total to reach the total.
3. Select the run to open it. Read the Latest score hero and the per-scenario table.
4. Work down the failures. Each row states which half failed — the root cause or the action. A root-cause failure usually means missing knowledge; write the runbook and activate it. An action failure requires checking the proposed operation, target evidence, opted-in connection and expected answer; adding a legacy route is not a prerequisite for prepared plans.
5. Where the automatic grade is simply wrong, select Override on the row, type why (the note is audited), then select the row's one button: Mark pass on a failing row, Mark fail on a passing one.
6. After changing a model, a prompt or a skill, open the same run and select Re-run same scenarios. This replays the exact scenario set as a fresh run.
7. Open the new run and read its gate block.

You should see it read Gate: PASS, with the run's pass rate against the same-dataset baseline and the floor (80% by default) beside it, and no regressions listed. Named regressions and improvements are listed by scenario id. With no earlier run over the same dataset there is no baseline, and the gate is the floor alone. The same holds across a grader upgrade: a case that flips between two grader versions moved with the grading rules, not the agent, so the automatic baseline only considers runs graded by the same version, and naming one graded by another returns a 409.

If a run is still marked running after a restart, open it and select Resume (after a crash). Only the scenarios missing from its results re-run; resuming a finished run is refused with a conflict. A re-run inherits the source run's access mode, so "we changed the model, re-test" never silently changes what is being measured as well.

Export a run, or throw it away

Export report renders the run as a markdown artifact: the headline number, what was measured (mode, dataset, agent fingerprint), the failures with the expectation beside them, and any adjudicated overrides. It is rendered by code, not by a model, so a customer-facing number has nothing in it to hallucinate. It lands on the Artifacts page, PII-scrubbed on write like every other artifact. Needs benchmark.view.

Delete removes the run, its per-scenario results and the sandbox investigations it produced. Needs benchmark.manage. Benchmark output is test output — unlike incidents and the audit trail it is meant to be disposable — so this is a real delete and it cannot be undone.

What a score means

The grader evaluates the root-cause and action claims the case actually supplies:

• Root cause: for a service-name truth, the asserted root_cause_service decides when the wrap-up carries that field. An empty field is no answer; a prose mention cannot replace or outvote it. Wrap-up schemas without the field (other pillars) are read from their root-cause prose. A free-text truth is matched against the field plus prose: an affirmed mention, or at least half of its significant tokens (at least four characters, plus selected three-letter technical terms such as DNS, TLS and CPU). An asserted root_cause_service blaming a declared must-not-blame service fails the case.
• Action: match the case's action keys in the proposed fix or remediation plan's steps, commands and manual commands. Without action keys, match the expected action as a full string or at least half of its significant words; one stray word is not a match. Hyphens and underscores act as separators, and regular plurals count as their word. This is an action-class match, not an exact-command comparison.

A truth that states only a service or only an action is graded on that half; when it states both, both must match. A declared suspect-change truth adds its own check. Each case shows which check failed. Each case scored by this grader records its grader version (v2 from 0.1.109). Certificates name it when relevant result rows carry a version stamp; legacy unstamped results from this grader count as v1. Other graders may have no such row. Compare uses matching versions. The model report uses the newest grader present: v1 is omitted when v2 is present; v1-only results remain shown.

An operator override wins over the automatic grade and badges the row SME override. That is what makes an aggregate pass rate defensible in a review — a machine-graded number nobody was allowed to correct is not evidence.

Three identities matter when comparing results: the fingerprint identifies the agent’s model, prompt and skill pack; the dataset hash identifies the scenario set; the grader version identifies the matching rules. Keep all three with a score. A changed pass rate across a changed catalogue or grader does not isolate a change in agent quality.

Use cases

Deciding whether to trust the agents — engineering manager

Before switching an alert route to the incident agent, the manager exports 60 closed incidents, uploads them with Upload golden set, checks the run's scenario count matches the file, and reads the per-scenario failures. Two failures are grading mistakes and get an audited Override; four are genuine knowledge gaps and become runbooks. Re-run same scenarios after the runbooks are active shows the gate block flip to Gate: PASS with two named improvements and no regressions. That run, its fingerprint and its overrides are what goes in the change record.

Troubleshooting

Symptoms specific to this page. Sign-in problems, missing nav entries in general, stuck or queued investigations, and $0.0000 costs are in the Troubleshooting chapter.

Next steps

• Test Lab — live fault injection, the lab box and how Live and Replay differ.
• Set up the Test Lab — provision the lab before a live run.
• Feedback — the human scores a back-test can filter on.
• Artifacts — where an exported report or a certification lands.
• Knowledge base — a root-cause failure usually means a missing runbook.
• Knowledge, x-ray and insights — the hub for every page of this group.