# Connecting the LLM gateway

> Three deployment settings that give the platform a model, worked gateway configurations, how to verify them, the built-in mock model, where model prices come from, and rotating a key safely.

_How-to · Updated 2026-10-17 · Describes release 0.1.164_

## What this is for

Every agent in this product — incident investigations, insights chat, estate x-ray, benchmarks — is one loop of `tool call → tool result → next decision`, driven by a large language model. The product ships with **no model configured**. Until a platform operator points it at a model endpoint, every one of those surfaces still works, still clicks through end to end, and still produces a report — from a deterministic **mock model** that invents nothing new and proves nothing about your estate.

This chapter is the one piece of configuration that turns the demo into a product. It is deployment-level work: three environment settings on the control plane (and on any runner processes), applied by whoever owns your deployment. There is no in-product screen that sets them, and no RBAC permission that grants them — an operator with shell or chart access to the deployment is the person who does this.

Once a model is configured, **which** model does which work is a separate, in-product decision made on **Platform defaults** and on each tenant's **AI models** page. That is covered in [Platform configuration](/help/ops-platform.md); this chapter covers getting a working endpoint underneath it.

## Concepts

**Gateway.** Any HTTP endpoint that speaks the OpenAI chat-completions API. A litellm proxy, OpenRouter, an enterprise LLM gateway, a vLLM server, Ollama's compatibility endpoint — the platform does not care which, and has no special code for any of them.

**Model string.** A litellm model identifier such as `openai/gpt-4o` or `bedrock/amazon.nova-pro-v1:0`. The prefix before the first `/` selects the provider; the rest is the model id that provider understands.

**Completion client.** The only LLM dependency in the product is **litellm**, installed as the `agent` extra (the shipped container image installs it). Our agent loop is our own code; litellm is nothing but the multi-provider HTTP client underneath it.

**Tier.** A category of work — `premium`, `reasoning`, `standard`, `light` — that a model string is mapped to. Tiers are configured in the product; the settings in this chapter are what tiers resolve *to* when nothing more specific is set.

## The three settings

Every setting in this product is read from the process environment with a `AIOPS_` prefix. These are the real variable names:

| Variable | What it does | Empty means |
|---|---|---|
| `AIOPS_MODEL` | The litellm model string used for every unit of work that no tier map or override resolves. | The deterministic mock model runs. |
| `AIOPS_API_BASE` | The base URL of an OpenAI-compatible gateway, e.g. `https://llm-gateway.example.com/v1`. Passed to litellm as `api_base` on every call, and used for gateway model discovery. | No gateway; litellm routes by the model string's provider prefix alone. |
| `AIOPS_API_KEY` | The credential for that endpoint. Passed to litellm as `api_key`. | No key is sent. |

Two related settings belong in the same change:

| Variable | What it does |
|---|---|
| `AIOPS_FALLBACK_MODELS` | Comma-separated model strings litellm fails over to when the primary errors (for example a gateway 502). Usage from a fallback is booked under the fallback's id, not the primary's. |
| `AIOPS_PRICE_PER_1M_INPUT_USD` / `..._OUTPUT_USD` | Fallback token prices in USD per million tokens, used only when litellm returns a cost of zero. See *Cost, tokens and the pricing map*. |

All of these are read **once, at process start**. Changing any of them requires restarting the process that reads them — there is no reload endpoint and no hot path that re-reads the environment.

## Where you set them

### Docker Compose

The compose file maps unprefixed names from your `.env` file into prefixed container environment:

```yaml
environment:
  AIOPS_MODEL: ${AIOPS_MODEL:-}
  AIOPS_API_BASE: ${AIOPS_API_BASE:-}
  AIOPS_API_KEY: ${AIOPS_API_KEY:-}
```

So in `.env` you write `AIOPS_MODEL=...` without the prefix, because Docker Compose — not the application — reads that file for substitution. If you ever run the application directly against a `.env` file, the prefix is mandatory there: a bare `AIOPS_MODEL=` line is invisible to the app.

### Helm chart

The chart splits them deliberately. The model string and the endpoint are non-secret and land in a ConfigMap; the key is a Secret:

```yaml
config:
  AIOPS_MODEL: "openai/gpt-4o"
  AIOPS_API_BASE: "https://llm-gateway.example.com/v1"
  AIOPS_FALLBACK_MODELS: ""
secrets:
  AIOPS_API_KEY: ""
```

Set `secrets.AIOPS_API_KEY` with `--set-string` from your CI secret store, or point `existingSecret` at a Secret whose keys match these names. Anything the chart does not model — an extra provider variable, for instance — goes in `extraEnv` as a `{name, value}` entry.

### CI pipeline

The shipped pipeline passes the key as a masked variable and the model and endpoint as plain config on the Helm upgrade. Keep the key in the deployment environment only; it does not belong in a values file that is committed.

### A note on variables the app does not model

The application's settings object ignores environment variables it does not recognise, and litellm reads its own provider variables straight from the process environment. That is how provider-specific settings such as `AZURE_API_VERSION` and the standard `AWS_*` variables reach litellm even though this product never declares them.

## How a model string routes

litellm resolves the provider from the prefix of the model string:

| Model string | Where the call goes |
|---|---|
| `openai/gpt-4o` with `AIOPS_API_BASE` set | Your gateway, over the OpenAI chat-completions API, with the id `gpt-4o`. |
| `openai/anything/your-gateway-calls-it` | The same, with that id verbatim. The `openai/` prefix means *speak the OpenAI protocol*, not *call OpenAI*. |
| `bedrock/amazon.nova-pro-v1:0` | AWS Bedrock directly, using the standard AWS credentials in the environment. |
| `azure/<deployment-name>` | Azure OpenAI, using `api_base`, `api_key` and an API version. |
| `gpt-4o` (no prefix) | **Fails to route.** A bare id has no provider. |

**This is why the `openai/` prefix matters.** Gateways advertise their own bare ids — `gpt-4o`, `claude-sonnet`, `internal/llama-70b`. When the platform reads a gateway's model list it prefixes each id with `openai/` so the value in the dropdown is directly usable as a model string. The prefixing is idempotent: an id that already starts with `openai/` is left alone. If you type an id by hand anywhere in the product, you must add the prefix yourself.

Every call the platform makes carries a 300-second timeout and two litellm-level retries, with the OpenAI SDK's own slow backoff disabled — a flaky model fails over to `AIOPS_FALLBACK_MODELS` in seconds rather than sitting in per-attempt backoff for minutes.

## Worked configurations

### An OpenAI-compatible proxy

The common enterprise case: one gateway in front of several vendors, one key, one allow-list.

```bash
AIOPS_MODEL=openai/gpt-4o
AIOPS_API_BASE=https://llm-gateway.example.com/v1
AIOPS_API_KEY=sk-your-gateway-key
AIOPS_FALLBACK_MODELS=openai/gpt-4o-mini
```

The base URL must be the one whose `/models` and `/chat/completions` paths are valid — usually the one ending in `/v1`. This is the only configuration in which the gateway model dropdowns fill in, because model discovery is a `GET` against `AIOPS_API_BASE` + `/models`.

### Azure OpenAI

Azure addresses a *deployment name*, not a model name, and needs an API version:

```bash
AIOPS_MODEL=azure/your-deployment-name
AIOPS_API_BASE=https://your-resource.openai.azure.com
AIOPS_API_KEY=your-azure-key
AZURE_API_VERSION=2024-10-21
```

`AZURE_API_VERSION` is read by litellm from the process environment; add it through `extraEnv` on the chart or the service environment block in compose. Azure OpenAI does not serve an OpenAI-style `/models` list at this base URL, so the model dropdowns stay empty and you enter ids under **Custom model id…**. That is cosmetic; routing works.

### AWS Bedrock behind a gateway

Put Bedrock behind your gateway and it is an OpenAI-compatible proxy like any other — this is the configuration to prefer, because it keeps AWS credentials out of the platform entirely:

```bash
AIOPS_MODEL=openai/bedrock-claude-sonnet
AIOPS_API_BASE=https://llm-gateway.example.com/v1
AIOPS_API_KEY=sk-your-gateway-key
```

Calling Bedrock directly is also supported. Use a `bedrock/` model string and give the process standard AWS credentials; leave `AIOPS_API_BASE` empty, because it would be applied to the Bedrock call as well.

```bash
AIOPS_MODEL=bedrock/amazon.nova-pro-v1:0
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION_NAME=ap-southeast-1
```

litellm looks for `AWS_REGION_NAME` first, then `AWS_REGION`. Set both to the same region: agent runners receive only `AWS_REGION`, and the platform refuses to price a model from the AWS Price List while the control plane and its runners would call different regions. Direct Bedrock has no `/models` list either, so the dropdowns stay empty.

Behind a gateway, state the region on each Bedrock entry of the gateway's config (`litellm_params.aws_region_name`): it is how the platform knows which region's AWS price applies — see [Official prices: the AWS Price List](#official-prices-the-aws-price-list).

### A regulated workspace: an in-region Bedrock route

A workspace that must keep its model traffic inside a geography needs a gateway alias whose backing model is a regional inference profile, and a residency policy that refuses every other route. The shipped gateway configuration carries one such alias:

```yaml
model_list:
  - model_name: bedrock/claude-3-5-sonnet-apac
    litellm_params:
      model: bedrock/apac.anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: ap-southeast-1
```

An `apac.` profile is served inside the APAC regions only: the entry point is the stated region, and Bedrock may fulfil the call in another APAC region, never outside the geography. It does not promise single-region processing. The platform also accepts a plain in-region model id when the effective region is known. Confirm current model access and profile availability in the account before publishing an alias; an entry that errors is worse in the tier dropdown than one that is absent.

1. Publish the alias through the normal reviewed change to the gateway configuration. Confirm from the gateway's `/model/info` that the entry reports its backing model and `aws_region_name`; an alias name alone is no residency evidence, and an entry that pins `model_id` is called at that id, so inspect that target too.
2. Fetch and review the alias's prices on **Platform defaults → Model price map**. Do not give an unresolved alias an invented price.
3. In the workspace, pin every tier the workload uses to the allowed route on **AI models → Model tiers**, and review any pillar override or fallback that could change the effective model.
4. On **Guardrails & cost → Model residency**, turn on **Singapore regulated (MAS/PDPA)** and read the saved policy. Under it, global profiles, other geographies, non-Bedrock routes and routes whose region cannot be placed are refused before a model call.
5. Run one bounded, authorized test and read the **Model route recorded** or **Model denied by route policy** audit row with its alias, backing model and region. A successful answer alone proves nothing about the route. Keep the policy and the routing evidence together.

This is evidence of the model route the platform used, not legal certification. The mock model makes no model call and proves no regional provider path. A Bedrock model that refuses prompt-cache markers is retried without them and that refusal is remembered for the process lifetime; the retry is not a reason to change region. Workspace guide: [Models, tiers and routing → Set it up like the demo](/help/models-tiers-and-routing.md#set-it-up-like-the-demo).

### A self-hosted vLLM or Ollama endpoint

Both expose an OpenAI-compatible surface, so both are the proxy case with a local URL. vLLM's id is the `--served-model-name` you started it with:

```bash
AIOPS_MODEL=openai/your-served-model-name
AIOPS_API_BASE=http://vllm.internal.example.com:8000/v1
AIOPS_API_KEY=not-used-but-set-something
```

For Ollama, use its compatibility endpoint and the tag you pulled:

```bash
AIOPS_MODEL=openai/qwen2.5:14b
AIOPS_API_BASE=http://ollama.internal.example.com:11434/v1
AIOPS_API_KEY=ollama
```

Set a non-empty key even when the server ignores it; some clients reject an empty bearer. Both servers do publish `/models`, so the dropdowns fill in.

**Before you commit to a self-hosted model, confirm it does tool calling.** The agent loop hands the model a tool schema on every step and acts on the tool calls it returns. A model that cannot emit tool calls does not fail loudly — it answers in prose, the loop sees no tool call, and the run finishes immediately with a root cause and **no evidence and no tool timeline**. The same class of problem shows up at the final step: the wrap-up call asks for a structured JSON object, and if the model or gateway ignores that request the platform salvages what it can and otherwise records the raw text as the root cause with a confidence of `0.0`.

## Verify the gateway took effect

Do all four checks. Each proves a different layer, and the first three can pass while the fourth fails.

Requires superadmin for steps 1–3; a tenant member with `report.view` for step 4.

1. Restart the control plane after applying the settings, and confirm it came up.
2. Open `/ops/platform` (**Platform defaults** under **Config**). The four tier pickers should list your gateway's model ids, each prefixed `openai/`. **This proves the endpoint and the key are correct** — the list is fetched live from `AIOPS_API_BASE` + `/models` with `AIOPS_API_KEY` as a bearer. If the pickers offer only **Custom model id…** and the page reads *"Gateway model list unavailable (…)"*, the message in brackets is the actual failure; `no gateway configured` means `AIOPS_API_BASE` is empty in the process you restarted.
3. Trigger one investigation in a tenant and open the incident. The investigation card's subtitle should name **your model id**, not `mock`. That proves routing.
4. On the same card, read the token and cost figures. Non-zero tokens prove a real call happened. A non-zero dollar figure additionally proves the model id is priceable — see the next-but-one section if it stays at `$0.0000`.
5. Open **Chat** in the tenant and ask something about your estate. A real answer varies with the question. The mock model always answers with the same sentence shape: *"The system currently reports N incident(s) (source: sys_incidents)."*

![Platform defaults: the tier pickers populate from the configured gateway's model list, each id prefixed openai/. A populated dropdown is the proof that AIOPS_API_BASE and AIOPS_API_KEY are correct.](/help/media/ops/ops-platform-defaults.png?v=0.1.132)

There is no page anywhere in the product that displays the configured `AIOPS_MODEL`, `AIOPS_API_BASE` or `AIOPS_API_KEY`. The dropdown filling in is the closest thing to a status indicator, and it is the check to teach your operators.

## The mock model

When nothing resolves to a model string — no override, no tier map, no `AIOPS_MODEL` — the platform selects a **deterministic scripted client** instead of failing. It makes no network calls and needs no credentials.

It exists so that a fresh deployment is demonstrable and testable end to end before anyone provisions an LLM budget: correlation, the tool timeline, the write refusal, approvals, the root cause analysis block, the x-ray read-only denial and the artifact all render from real product code paths with a scripted brain behind them. It is also what makes the test suite hermetic.

### How to recognise a mock run

| Signal | What you see |
|---|---|
| Investigation card | The model reads `mock`. |
| Tokens | Exactly 10 prompt and 5 completion tokens per loop call. |
| Cost | `$0.0000` on investigations and chat. |
| Root cause | Begins `Mock root cause for:` followed by the first 80 characters of the prompt. |
| Evidence | The single entry `mock tool output`. |
| Proposed fix | Always `restart the failing deployment`, with the command `kubectl rollout restart deployment/demo-service` and confidence `0.9`. |
| Chat | The `sys_incidents` count sentence, every turn. |
| Benchmarks | Runs are labelled **mock runner**. |

### Why every demo root cause analysis looks identical

Because the wrap-up is not generated — it is a fixed dictionary with your prompt's first 80 characters pasted into one field. Two different incidents produce two root causes that differ only in that fragment. If a customer says "the analysis is always the same", that is not a model quality problem; that deployment has no model configured.

### What the mock does not prove

- **Nothing about accuracy.** The scripted responder cannot match ground truth, so the platform deliberately does not run benchmarks through the real engine when no model is configured — it uses a ground-truth echo runner and labels the result honestly. An 80% pass rate from a mock deployment measures the pipeline, not the agents.
- **Estate x-ray spend is fictional.** The mock x-ray client reports a flat `$0.50` per call, so a mock scan accumulates a plausible-looking cost and can even hit its budget. Those dollars were never spent.
- **The gateway itself is untested.** A mock run passing tells you nothing about credentials, network reachability or tool-calling support.

## The default tier map and what a tenant inherits

`AIOPS_MODEL` is the floor, not the routing policy. Work is routed by tier — `premium` for x-ray scans, `reasoning` for the root cause wrap-up, `standard` for the investigation loop and chat, `light` for bulk work — and each tier maps to a model string.

A tenant that has never configured its own tier map **inherits the platform default map** you set on **Platform defaults**. A tenant that sets even one tier owns the whole map: the two are never merged, which keeps a tenant's spend predictable. A missing tier resolves to the nearest configured tier above it, then below, so setting only `standard` gives a working four-tier platform.

Two consequences worth knowing before you configure anything:

- **The tier map is the routing policy; `AIOPS_MODEL` is what remains when no map applies.** With one model and no map, every tier is that model and mid-run tier escalation has nowhere to go — the escalation ladder needs at least two distinct configured models to have a step to take.
- **In a `remote` deployment the runners follow the same map, from 0.1.117.** An api-mode runner receives the control plane's effective map with every job and a db-mode runner re-reads it each time it polls, so a tenant with no map of its own runs on the platform default there too, not on the runner's `AIOPS_MODEL`. A runner image older than 0.1.117 never sees the map and runs such a tenant on its own `AIOPS_MODEL_TIERS_DEFAULT`, else on `AIOPS_MODEL`.

The full resolution order, how runners receive the default map and its environment-variable form are in [Platform configuration](/help/ops-platform.md) — do not maintain a second copy of that chain.

## Cost, tokens and the pricing map {#cost-tokens-and-the-pricing-map}

Per call, the platform records prompt tokens, completion tokens, cached reads, cache writes and a dollar cost, and adds them to the run's row. Spend is booked in three places:

| Where | What it holds |
|---|---|
| The investigation row | Running totals for the whole run, plus a **per-model split** so a run that escalated mid-way credits each model with the work it actually did. |
| The x-ray scan row | Per-scan totals aggregated between phases, checked against the scan's budget every step. |
| The tenant report and chat | The report's *Cost per analysis* tile (in its AI spend appendix), and the `sys_costs` chat tool's per-pillar and per-model breakdown. |

Every booked figure also records **where its price came from** — `price_source` on each model's line of the run (`model_usage[model]`), and `price_ref`, the card entry that priced it:

| Source | What priced the call | Trusted |
|---|---|---|
| `aws-price-list` | The AWS Price List, read for a Bedrock model by **Fetch list prices** or the daily sync | yes |
| `models-dev` | The publisher's own list price as models.dev publishes it, for every model **not** served through Bedrock, reached through a dated mapping | yes — a third-party dataset, named as such |
| `manual` | A price a superadmin recorded with a source note — a contract, a quote, a pricing page; `0 / 0` = no token charge | yes |
| `estimate` | A trusted price that cannot state the whole bill for this call: a models.dev gap (DeepSeek peak hours, a reasoning rate), a prompt over the model's long-context threshold, a 1-hour cache write | no — estimate |
| `unattributed` | The deployment's own map (`AIOPS_MODEL_PRICES`, or the map saved before price history existed): a rate whose source nobody recorded | no |
| `litellm-estimate` | Nothing on the card: the model library's own price table (platform gateway only) — through a LiteLLM gateway, the gateway's copy of it (`price_ref` `gateway x-litellm-response-cost`) | no — estimate |
| `flat-estimate` | Nothing anywhere: the `AIOPS_PRICE_PER_1M_*` fallback rates (platform gateway only) | no — estimate |
| `byo-card` | A workspace's own typed rate card — runs **before** this release only (it was retired at the upgrade) | — |
| `openrouter-estimate` | A gateway-listed price adopted before this release; no longer written | no — estimate |

A run priced from an estimate still counts against budgets and the per-run cap — an estimate is better than `$0` for a cap — but every cost surface says which part of a figure is an estimate and why: **Billing** in this console, a workspace's **Usage & billing**, an incident's Spend card and the report. A run booked before price sources were recorded is read as an estimate when its model's line carries the old fallback flag and money.

**The model library's table is the estimate, and it is pinned.** litellm prices a call from a community-maintained table. Unpinned, it downloads that table from GitHub every time a process starts, so one release booked different estimates depending on which pod reached GitHub — measured on 13 October 2026: Claude Sonnet 5.5 at `$0` from the bundled copy, an `apac.` Claude Sonnet 5 at the global $2.00 instead of the regional $2.20, an `apac.` Nova Pro at $0.84 per 1M where AWS charges $1.08 from Singapore. Both images set `LITELLM_LOCAL_MODEL_COST_MAP=True`: an estimate is the table the release shipped, labelled with the litellm version that produced it. A call through a LiteLLM gateway is the exception: litellm books the cost the gateway returns in its `x-litellm-response-cost` header, computed from the gateway image's own table (which moves with that image and what it downloaded), so its `price_ref` reads `gateway x-litellm-response-cost`, never this release's version. The platform never writes the card into that table: every client books from the card itself, and a registration would outlive the card — a removed price kept booking until a restart, and another gateway's model sharing a bare id with a card entry booked the card's rate as the library's estimate. A gateway id litellm does not know at all prices at `$0` — see [Set the fallback rates](#set-the-fallback-rates).

### Official prices: the AWS Price List {#official-prices-the-aws-price-list}

For a model the gateway serves from AWS Bedrock, the platform reads the price AWS publishes — the public Price List files, no credentials, no AWS account:

1. **Alias → backing model and region**, from the gateway's own `GET /model/info`: `litellm_params.model` and `litellm_params.aws_region_name`. An entry that pins `litellm_params.model_id` is called at that id, so that is the target priced (and judged by a model route policy): a `global.` profile pinned there bills the global rows; an application inference profile or provisioned-throughput ARN hides the model and stays unpriced from AWS. **The gateway entry must state `aws_region_name`** — the platform never assumes the region the gateway pod happens to run in. With no gateway (`AIOPS_API_BASE` empty) and a `bedrock/…` model, the region is the control plane's `AWS_REGION`; it is refused when `AWS_REGION_NAME` says otherwise, because runners only receive `AWS_REGION`.
2. **Backing model → price-list product**, through a reviewed table (`BEDROCK_PRODUCTS` in `app/aiops/price_sources.py`) keyed by the exact Bedrock base id and the price list's exact model name. Claude Sonnet 5 and Claude Sonnet 5.5 share every AWS usage type and differ in the cache-read rate ($0.20 and $0.10 per 1M), so only the exact name may select rows. A model the table does not name stays unpriced from AWS and is reported; it is never matched by a similar name.
3. **Inference profile → rows.** A `global.` profile takes the global rows; a geographic profile (`apac.`, `us.`, `eu.`, …) or an in-region model id takes the regional rows of the **source** region — the region the gateway calls from. `AmazonBedrockFoundationModels` prices Anthropic and the other Marketplace-billed models; `AmazonBedrock` prices the Amazon models (Nova, listed per 1K tokens and converted to per 1M).
4. **Rows → rates.** Only the standard on-demand text rows: input, output, cache read and the 5-minute cache write. Batch, flex, priority, latency-optimized, long-context, 1-hour cache-write, custom-model and provisioned-throughput rows never match. A rate the list does not state stays absent (Claude 3.5 Sonnet v2 has no cache rows in ap-southeast-1). Two different prices for the same kind price nothing. A `$0` input or output price is never applied unattended.

What does not resolve stays unpriced and is listed with its reason under **Needs attention** on **Platform defaults → Model price map**: *the gateway entry for … states no aws_region_name*, *no reviewed AWS mapping for …*, *the gateway balances … across deployments with different models or regions*, *no regional on-demand input and output price …* (a model sold only through the global profile in that region), or *the platform gateway does not route …*. An alias the gateway serves from Bedrock is **never** priced from models.dev, even when `BEDROCK_PRODUCTS` lacks its model: add the reviewed row, or record a manual price citing the AWS list. Azure OpenAI has no price source yet; enter its price manually.

**Fetch list prices** runs the read now (`POST /api/aiops/admin/model-prices/sync`, superadmin with an admin-scoped token) — both parts: the AWS Price List here and [models.dev](#publisher-list-prices-models-dev) below. The control plane repeats it daily for the models in use — booked in the last 30 days, or configured as a tier, `AIOPS_MODEL` or a fallback — on the scheduler leader only: `AIOPS_PRICE_SYNC_INTERVAL_S` (default `86400`; a failed attempt retries after an hour; `0` turns both parts off for an install without outbound HTTPS to `pricing.us-east-1.amazonaws.com` and `models.dev`). Per Bedrock model the result reads *Added*, *Changed*, *Unchanged*, *Kept the manual price*, *Kept removed*, *Skipped ($0 on the list)*, *Price list unreadable*, *Not mapped* or *No official source*.

- A new or changed price is recorded **from that moment**, with one `platform.model_price` audit row; an unchanged one writes nothing. A price whose numbers match but whose product changed — an alias re-pointed from Claude Sonnet 5 to 5.5 — is a new entry.
- A **manual** price outranks the list price: the sync keeps it, and keeps a model you removed, unless you name the model — **Replace with the list price** beside a kept manual price.
- A download that fails changes nothing: the card keeps its last good prices and the result says the price list could not be read. A gateway whose `/model/info` cannot be read (restarting, say) fails the read the same way — every Bedrock alias is reported *Not mapped* — and it is retried within the hour, not the next day.
- Nothing is re-priced: booked history moves only through [the re-book](#re-price-booked-history).

After you change the gateway's model list, press **Fetch list prices**: until the next read, an alias re-pointed to another model books at the old model's price.

### Publisher list prices: models.dev {#publisher-list-prices-models-dev}

No model publisher offers a price API. For every model **not** served through Bedrock, the platform reads [models.dev](https://models.dev) (`https://models.dev/api.json`), an open, MIT-licensed community dataset of model prices — and only each **publisher's own** entry in it: OpenAI, Anthropic, Google (Gemini API), DeepSeek, Z.ai, Mistral, xAI, Moonshot AI, Alibaba Cloud (international) and Meta. Resellers and hosts (OpenRouter, Azure, Amazon Bedrock, Vertex), subscription plans listed at `$0` (`*-coding-plan`) and the CNY price lists are never read — nor another publisher's model a publisher's own block resells (models.dev lists GLM, DeepSeek and Kimi models under Alibaba and Mistral too; the model's canonical publisher decides, so only Z.ai's, DeepSeek's and Moonshot's own entries price them). On 13 October 2026 it matched the publisher pages on 28 of 30 models; the two misses were DeepSeek's off-peak-only rows (see gaps below).

- **Catalogue.** Each priced publisher model becomes a dated entry under a reserved key — `models.dev/openai/gpt-5.6-sol` — append-only like every card entry: a change is a new entry from the moment it is read, never an edit. The entries never ship as card keys; **Publishers and workspace endpoints → Publisher catalogue** lists them with their history, gaps and the publisher's pricing page.
- **Mappings.** An alias reaches its publisher's price through a dated mapping: *from this date, alias X is publisher model P*. A mapping is resolved when a price is read, so one publisher change re-prices every alias bound to it from its date. Every spelling runs book under is mapped on its own (`openai/glm/glm-5.3` and `glm/glm-5.3`).
- **Only the reviewed router table maps unattended:** `cx/` → OpenAI (a trailing `-review` is the same model on another quota), `glm/` → Z.ai, `ds/` → DeepSeek, `cc/` → Anthropic, applied to the backing the gateway's `/model/info` reports (else to the alias). `cx/gpt-5.3-codex-spark` has no publisher price — OpenAI publishes none and models.dev's copied figure is refused. A mapping the sync recorded follows a re-pointed backing: to another model the table names, it is re-mapped from that read; to one with no publisher price behind the table — outside it, not listed by models.dev yet, no publisher price, served from Bedrock, or balanced over deployments that serve different models — it ends from that read and the alias is listed as not mapped (it never keeps booking the old model's price). A mapping an operator recorded is never changed, only reported when the table disagrees. While the gateway's `/model/info` cannot be read, the table changes nothing.
- **Everything else is mapped by you**, from the catalogue: **Map** beside a model under *Needs attention* offers suggestions first — high confidence: the router table, the gateway's own `model_info.base_model`, a native backing (`anthropic/…`, `gemini/…`); low confidence: the model the endpoint said served the calls (recorded on every usage line — text a customer's endpoint or agent reports) and a match by name alone — then a search grouped by publisher. The dialog opens on a high-confidence suggestion only; a low-confidence one is listed for you to check and pick. **Map all suggested** applies the high-confidence ones in one confirmation; a low-confidence one is never applied in bulk. A mapping may not start before its model's first catalogue entry; a run before that is priced by a manual entry, if at all. `POST /api/aiops/admin/model-prices/mappings` records the same rows (a note on each; `if_current` refuses a stale page with 409).
- **Held changes.** A change that moves any rate beyond **2× either way** — a unit slip, a batch rate published as the standard one (models.dev showed GPT-5.6 Sol at its batch rate for days in August 2026) — waits under *Needs attention* until you press **Confirm** (`confirm` on the sync). So does a first mapping that would move an alias's current price that far, or off a `$0`. For a publisher model, a cache rate that appears or disappears counts by what it does to the bill (absent = billed at the input rate): models.dev dropping GPT-5.6 Sol's cached-read rate would bill cached tokens ten times higher, so it waits too. **Confirm** applies the change it showed you: if models.dev publishes another value before you press it, that value waits again with its own numbers. A held change stays listed through a failed read.
- **Gaps are flagged, never filled in.** DeepSeek bills twice its listed rate at peak hours; models.dev lists the off-peak rate, so DeepSeek lines book as `estimate`. A reasoning rate that differs from the output rate books as `estimate`. A long-context tier (OpenAI above 272K prompt tokens, Gemini Pro and xAI above 200K, Claude Haiku 5.5 above 100K, three Alibaba models above 32K) is kept as a threshold: a call over it books at the base rate as `estimate`. Dated prices models.dev cannot state — GPT-5.6 Sol's promotional price "at least through November 21, 2026", Mistral Large 4's sale price, Gemini 3.8 Flash's 1 January 2027 rise — are a reviewed watch list: from the review date *Needs attention* asks you to check the publisher's page. Batch, flex, priority and regional uplifts are never requested by the platform; the standard global rate is the one it books.
- **Safeguards, as for the AWS part.** One fixed URL, no redirects, TLS, 16 MB and 40 seconds, a conditional read (`If-None-Match`), a shape check; any failure keeps the last good catalogue and every mapping, and the hourly retry follows. A `$0` price is never applied, a manual price is never overwritten, nothing is re-priced. `AIOPS_MODELS_DEV_PRICES=false` turns this part off (the AWS part keeps running).
- **What a customer sees:** *OpenAI list price · gpt-5.6-sol · models.dev, read 14 Oct 2026*, and on the statement one sentence naming the sources that priced the month — models.dev and its licence when it priced any. A DeepSeek or long-context estimate is said as a list price that states part of the bill, never as a missing price.
- **Back-dating** is the same human act as for an AWS price: models.dev follows a publisher change within days, so when the gap matters record a manual entry with the publisher's rates from the publisher's date (its note citing the publisher's page and the models.dev revision), press **Replace with the list price** so the mapping rules again from now, then re-book (dry run first).

### The card keeps its history

Every price is an entry with its source, its note, who set it and when, and the date it **applies from**. A save never rewrites an entry: a corrected rate is a new entry, and the database refuses an edit to an old one. Each entry says *this rate from this date on*, so the price in force for a model at any moment is the entry **recorded last** among those applied by then — a later save outranks an earlier one from its own date on, even one dated after it — with the deployment's own map underneath for models no entry covers. The deployment's map is rates only: a `source` written into an `AIOPS_MODEL_PRICES` entry is ignored, and its rates read *source not recorded* until someone records them. **Each run is priced at the entry in force when it started** — the cost split on every surface and the re-book alike — so a price change never re-prices the past on its own.

- **A manual price needs a source note** — the contract or the pricing page it comes from. The editor refuses a changed, new or removed price without one (`PUT /api/aiops/admin/model-prices` answers 400 naming the models), and a price that does not state both its input and output rate (`0` states a free model). **Record source** under *Needs attention* records the rate that prices the model now, unchanged, with your note — under the model id runs book with, wherever the deployment's map keys it (for example without the platform's `openai/` prefix).
- **Apply from** dates the edited prices (today by default, never in the future). The price applies from that date **on**, the present included, over every price recorded before it — an official AWS entry dated later too, which the daily sync then keeps as a manual price. A price applied from today leaves the runs that already happened as they were booked. To correct the past, apply it from when the rate was really in force — the editor suggests the edited models' first booked run — then re-book. To correct only a past period, save that period's rate from its start, then the rate that followed from its own date.
- **A save carries the card's version** (`version` from `GET`). When the card changed after the page loaded — the daily sync recorded a price, another administrator saved — the save is refused (409) and the page reloads the card: make the edit again. A full map saved over a newer card would otherwise remove the prices it does not hold, or write its stale rates over them as manual prices the sync then keeps.
- **A model left out of the map is removed from now** (an entry too, with the note saying why). An empty map is refused.
- **Back-dating an official price is a human act.** The daily sync starts an official price *now*: runs booked before it stay estimates until a superadmin records the price that applied then — a manual entry applied from when the alias began routing to that model, with a note citing the AWS publication or the publisher's page — and re-books. Like any save, that entry rules from its date on, the present included, so the sync would keep it as a manual price: press **Fetch list prices**, then **Replace with the list price** beside it, and the list price rules from now again while the manual entry covers the past. The platform keeps no per-run record of which model an alias meant on a past date, so it does not back-date on its own.
- **A mapped alias in the editor** shows its publisher's rates; editing them records a manual entry for the alias that outranks the mapping from its date. A row for a catalogue key (`models.dev/openai/gpt-5.6-sol`) records an override of that publisher model for every alias mapped to it.
- Only rates, the source, the entry id, the date and — for a models.dev price — the long-context threshold and gap codes travel to agent runners (customer-run private agents included). The note, the page, the revision and who set it stay on the control plane; a workspace sees the kind of price, its date and, for an AWS price, the region and publication date, for a models.dev price the publisher, model, read date and pricing page — never your note.

### Set the fallback rates {#set-the-fallback-rates}

The platform applies a per-token estimate whenever nothing priced a call that used tokens — no card entry and no litellm price:

```bash
AIOPS_PRICE_PER_1M_INPUT_USD=3
AIOPS_PRICE_PER_1M_OUTPUT_USD=15
```

Use the rates your gateway actually bills you. They apply to *every* unpriced model in the deployment, so on a multi-model gateway choose the rates of the model doing the bulk of the work. Calls priced this way are labelled `flat-estimate` and shown as estimates; they keep the caps working, they are not an invoice. Left at `0` the feature is off and an unknown id books `$0`.

### Cached prompt reads — price them explicitly {#cached-prompt-reads-price-them-explicitly}

An investigation replays its growing transcript on each step. The gateway may report **cached prefix reads** within its prompt tokens; their share and price depend on the gateway and workload. The platform records reported cached reads per model (`model_usage[model].cached_tokens`), and the incident Spend card shows their share. Use the rate your gateway actually bills rather than assuming a vendor ratio.

Their booked price depends on `cache_read_per_1m` on the card. This example is illustrative; enter your gateway's rates:

```json
"openai/gpt-5": { "input_per_1m": 1.25, "output_per_1m": 10, "cache_read_per_1m": 0.125 }
```

- `cache_read_per_1m` present → cached reads book at that rate.
- `cache_read_per_1m` absent → cached reads book at the **full input rate**. This is deliberate: a missing rate over-counts, and over-counting is the safe failure mode for a billing number. Confirm any correction against your gateway's rate card and invoice; the platform does not assume a discount.

**A missing rate is a visible signal, not a silent over-count.** Every surface that shows money — `GET /{tenant}/spend`, an investigation, an incident's Spend card, the statement — carries a `cost_breakdown`: the same dollars split into fresh input, cached reads, cache writes and output, at the rates that applied when each run ran, per model, with each model's price source and these lists: `unpriced` (models no entry priced — their booked figure passes through undecomposed), **`cache_rate_missing`** (priced, cached reads, no `cache_read_per_1m` — the tenant is told the figure is an **over-estimate**), `cache_write_rate_missing` (cache writes billed at the input rate) and `estimated` (booked money that came from an estimate). The card's models with no cache-read rate show as `cache read = input` on **Prices in force**.

A rate entered from today clears the banner for runs from today on; the runs already booked keep their split and their figure. A rate applied from an earlier date re-prices the split of the runs it covers on the next read, while the booked `cost_usd` on each run does not move until you re-book: the payload then comes back `rebook_needed: true` and the tenant's page says what was booked, what the rates now recorded for that time put the same tokens at, and that re-pricing history brings them together. `rebook_needed` is also set when the money agrees but a run booked as an estimate is now covered by an official or contract price — the re-book only re-labels it (an estimate the price itself carries, a models.dev gap or a long prompt, stays one).

The fields are accepted wherever a price is entered: the platform card (**Platform defaults → Model price map**, `PUT /api/aiops/admin/model-prices`, or the `AIOPS_MODEL_PRICES` environment variable) and a workspace endpoint model's own rates (**Map → Own rates**). models.dev states the cached-read rate (and the write rate where the publisher prices one), so a mapped model gets it without an entry; a `0` cache rate there is dropped (absent = input rate). DB-free agent runners receive the **platform** card from the control plane with every job — a workspace gateway reaches API-mode agents only when its owner explicitly shares it for agents, and then with its own card (the operator's mappings over the catalogue). Its key is resolved through a workspace-scoped virtual secret reference, never returned in the configuration payload, and a daily catalogue change never rotates that reference.

### Workspace endpoints (bring your own LLM)

For a model on a workspace's own endpoint, the billing price is the publisher's list price of the model it really is — never what the endpoint states: not a LiteLLM proxy's `x-litellm-response-cost` (their `$0`, or their price after their own discounts), not an OpenRouter-shaped `/models` price, not the rate card the workspace typed.

- **Mapped** models book their publisher's list price (`models-dev`) or the rates you recorded (`manual`); **No token charge** is your `0 / 0` with a written reason (self-hosted open weights, included by agreement, internal test).
- **Unmapped** models book `$0` with no source: every surface prints **Not priced yet**, the ops invoice shows the tokens under *Not priced — resolve before invoicing*, and *Needs attention* counts them. The flat fallback never applies to a workspace endpoint, so the per-run cost cap cannot see an unmapped model's calls until you map it; steps and the context budget still bound a run.
- **Who maps:** the platform operator only — **Publishers and workspace endpoints → Workspace endpoints** lists each workspace's models with their runs, tokens, money and the models the endpoint said it served, with **Map**, **Own rates** and **No token charge**. A lab box uses its parent's mappings. Each mapping lands a `platform.model_mapping` row in that workspace's audit log ("Model price set by the platform operator"), without your note.
- **The old rate card** a workspace typed keeps pricing exactly the runs before the upgrade (`byo_cards_retired_at`, stamped once at control-plane startup); a card saved later prices nothing. `PUT /{tenant}/llm-endpoint` still accepts `prices` and stores them, and its answer says the platform sets the prices; `GET` lists each endpoint model with its price status.
- After mapping, **Re-price history (dry run)** on the workspace shows what moves — the workspace's runs, those of every lab box on its endpoint and the Test Lab's, the runs the group counts; re-price it from there or with the API below (`include_lab_boxes`, `include_test`).
- Only the endpoint's own lines are listed there: the runs a workspace made on the platform gateway before it switched (or beside it) stay under the platform's models.

### Cache writes

Writing a prompt into the provider's cache is billed too: Anthropic models on Bedrock charge about 1.25 times the input rate for it (Claude Sonnet 5: $2.50 per 1M against $2.00 input), Amazon Nova charges nothing. The platform records cache-write tokens per model (`model_usage[model].cache_write_tokens`; they are part of the prompt tokens, so fresh input is prompt − cached reads − cache writes) and books them at `cache_write_per_1m`. With no write rate on the card they book at the input rate and the split flags `cache_write_rate_missing`, because the direction of the error depends on the provider. The AWS sync records the 5-minute write rate; a 1-hour cache write — the platform never asks for one — books at the 5-minute rate and marks the call an estimate.

### Re-price booked history {#re-price-booked-history}

Changing the card does not move what was already booked. **`POST /api/aiops/admin/rebook-costs`** re-prices history deterministically from the recorded per-model tokens, **each run at the card entry in force when it started**: `{"tenant_id"?, "since"?, "until"?, "models"?, "include_test"?, "include_lab_boxes"?, "dry_run": true}` — `models` moves only those model lines (every other line keeps its value and label), `include_test` also takes the Test Lab's runs (skipped otherwise; most of a lab box's spend is there), both recorded in the audit row; `include_lab_boxes` (with `tenant_id`) also takes every lab box that runs on that workspace's own endpoint — reports `count`, `changed` (runs whose money moved), `relabeled` (runs where only the price source moved — the first re-book after price history existed stamps `unattributed` on every run the deployment's map priced; that is bookkeeping, not money), `booked_usd / rebooked_usd` and a sample with each model's `from → to` source; `dry_run: false` writes the new `cost_usd` (run total and per-model split, with the price source) and lands one `billing.rebooked` audit row per tenant with both counts. Run the dry run first and read the sample. Models no entry priced at a run's start keep their booked value and label, benchmark runs (unless `include_test`) and runs still in flight are left alone. A workspace on its own endpoint re-prices through its own view — your mappings for its models (`prices_source: "byo"` per workspace) — while the lines its retired rate card priced keep their value and label. In every other workspace, a model line whose recorded provenance says a workspace endpoint booked it — `byo: true`, `byo-card`, or the library's estimate booked on a workspace gateway — keeps its value and label: a workspace that switched bring-your-own off keeps what its endpoint cost. Three numbers are frozen copies and do not move: a test-lab run's per-case metrics, an x-ray scan's `cost_usd` (summed from its phase runs once, at finish) and a validation recheck's `cost_usd` (no per-model split) — re-read the investigation rows for the corrected figure. A month the data-retention sweep has sealed is frozen too: its statement is read from the sealed row (**Platform defaults → Data retention default**), so re-booking changes neither the invoice nor the workspace statement for that month. Needs an admin-scoped token, like every `/admin` route.

### Reconcile the tokens with AWS

The money is only as good as the tokens. For Bedrock models, compare what the platform booked with what AWS counted — read-only, from the control-plane pod so it uses the platform's own AWS identity:

```bash
kubectl -n aiops exec deploy/aiops -- python -m app.aiops.price_sources reconcile --days 7
kubectl -n aiops exec deploy/aiops -- python -m app.aiops.price_sources reconcile --since 2026-10-01 --until 2026-10-08 --json
```

Per Bedrock model id and source region (`global.anthropic.claude-sonnet-5 @ ap-southeast-1`) it prints the platform's fresh input, cache reads, cache writes and output against CloudWatch `AWS/Bedrock` `InputTokenCount`, `CacheReadInputTokenCount`, `CacheWriteInputTokenCount` and `OutputTokenCount` (dimension `ModelId`), the drift and a verdict: **match** (within 2%), **aws-higher** (other callers share the account or region — expected on a shared gateway) or **ours-higher** (the platform booked tokens AWS did not count: investigate). Models that are not Bedrock, or have no region, are listed apart. It needs `cloudwatch:GetMetricData` on the control plane's role; without it the command names the missing permission and exits `2`. Nothing is written.

**On a customer's own AWS account (BYOC)**, reconcile the money in Cost Explorer: filter **Service** to the price list's service name (for example *Claude Sonnet 5 (Amazon Bedrock Edition)*, billed through AWS Marketplace, or *Amazon Bedrock* for Nova), group by **Usage type**, and compare the usage types each AWS price cites (shown on **Prices in force**) with **Usage & billing → By model** for the same UTC month. Quantity × the entry's rate is the list cost; a difference comes from other workloads in the account, a private price or credits (record a contract price as a manual entry — it outranks the list), or tax.

## Rotate a key without breaking in-flight runs

A key rotation is a process restart. What matters is what happens to work that is mid-flight when the process goes down.

### Embedded mode

The control plane runs agent jobs itself. On restart:

- Investigations from alerts, schedules and manual triggers that were `running` are **re-queued and resumed** from their committed transcript — the transcript is written at every step boundary, so a resume replays at most the one interrupted step. Each resume costs an attempt; the default allowance is three, after which the run fails and its incident is escalated.
- Estate x-ray scans in mission mode resume the same way, rehydrating their ledger.
- Chat turns, benchmark runs and OpenAI-facade calls are **not** resumable and are failed — the caller re-asks.
- Runs sitting in `paused_approval` are untouched and resume when a human decides.

Procedure. Requires deployment access; no product permission applies.

1. Mint the new key on the gateway and confirm the **old key is still valid** — an overlap window is what makes this safe.
2. Update `AIOPS_API_KEY` in the ConfigMap, Secret or `.env` that feeds the control plane.
3. Restart the control plane.
4. Open `/ops/platform` and confirm the model dropdowns still populate. That is the new key succeeding against the gateway.
5. Trigger one investigation and confirm it completes with your model id on the card.
6. Revoke the old key at the gateway.

You should see previously-running investigations continue and finish rather than appear as failures; the incident feed shows them advancing again within a sweep interval.

### Remote mode and per-tenant runners

With `AIOPS_EXEC_MODE=remote` the control plane only enqueues work; per-tenant runner processes execute it. **Restarting the control plane does not disturb a running job**, which is the point of the topology — but it also means the control plane's new key does not reach the agents. The runners hold their own copy.

How the new key reaches a runner depends on how runners are provisioned:

| Provisioner | Behaviour on a key change |
|---|---|
| `kubernetes` | The reconcile loop compares the running Deployment's environment against the desired environment. A changed model, endpoint or key is detected as **env drift** and the Deployment is patched. The strategy is `Recreate`, so the old pod gets its SIGTERM drain window (default 300 seconds) to finish in-flight steps; anything still running when it ends resumes on the next runner via its expired lease. |
| `docker` | **Image only.** The Docker driver does not diff environment, so a rotated key never reaches a running runner container. Remove the container so it is re-provisioned on the next reconcile tick. |
| None (compose `agents` profile, or hand-managed) | The runner's environment comes from wherever you defined it. Recreate the runner after updating it. |

In all three cases, an interrupted runner job is recovered by the same lease mechanism as an embedded restart: another runner steals the expired lease and resumes from the committed transcript.

Sequence for a remote deployment:

1. Confirm the overlap window at the gateway.
2. Update the key in the control-plane configuration **and** in the runner configuration.
3. Roll the control plane.
4. Roll or recreate the runners, one tenant at a time if you want to bound the blast radius.
5. Verify per tenant: trigger an investigation, confirm it completes on your model id.
6. Revoke the old key.

Update the gateway configuration used by private agents as well. A successful control-plane request does not prove that a remote investigation, scan or conversation can authenticate — hosted chat may succeed while runner investigations fail, and private-bound workspace chat runs on private agents. A hosted chat result does not validate a private agent’s model credential. Verify a real task through the relevant agent after rotation.

## Prompt caching and the prompt diet

Two platform defaults decide what one agent call sends. Both are environment settings on the control plane, read once at process start like the rest of this chapter, and both have a per-workspace override in the tenant's settings (`PATCH /api/aiops/{tenant}` with `{"settings": {...}}`, a JSON boolean, or `null` to return to the platform default). Remote runners follow the control plane, not their own environment: the control plane resolves both values into the configuration it ships with every job (`dbMode: api`), and copies both variables into the environment of every runner it provisions (`dbMode: db`, which builds from the database). A runner you run yourself — the compose `agents` profile — needs them set in its own environment too. An agent image older than 0.1.117 ignores them and sends the prompt it always sent.

| Variable | Workspace setting | Default | What it changes |
|---|---|---|---|
| `AIOPS_PROMPT_CACHE` | `aiops_prompt_cache` | `true` | Only the request wrapper. The model reads the same text. |
| `AIOPS_PROMPT_DIET` | `aiops_prompt_diet` | `false` | What the model reads: bounded always-on blocks. |

### Prompt caching

An investigation sends its whole conversation on every step: the system prompt (persona, the workspace's knowledge, the estate, the planning rules, the skill catalog), the tool list and every earlier step. With caching on, each loop call to a **Claude** route marks two places with `cache_control: {"type": "ephemeral"}`: the system message, which caches the tool list and the system prompt, and the newest message of the conversation. The next call reads everything up to that mark from the provider's cache at the cache-read rate (0.1× input on Claude Sonnet 5) and writes only what is new, at the cache-write rate (1.25× input).

- **Which calls.** Only loop calls that another call with the same prefix follows. The wrap-up, the best-of-2 arbiter, the self-check, the last possible loop step and any run of at most two steps (the correlation sweep, storm triage) send no marks: a marked call that nobody reads again is a 1.25× write for nothing.
- **Which routes.** Claude is judged on the model id the platform gateway routes an alias to — its `/model/info`, the same map a model route policy reads, refreshed on a timer — never on the alias name. A native route (`bedrock/…anthropic.claude…`, `anthropic/…`) is its own id. Every other route sends no marks: an OpenAI-compatible endpoint rejects request keys it does not know. That includes a tenant's own gateway (the platform never asks it for `/model/info`) and a gateway with no `/model/info`. `AIOPS_FALLBACK_MODELS` counts too: a failed call is re-sent to every fallback with the same body, so the marks go only when every fallback is a Claude route as well.
- **A provider that refuses the marks.** Bedrock answers a request with cache points for a model whose prompt caching never left preview — Claude 3.5 Sonnet v2, the model behind the in-region alias `bedrock/claude-3-5-sonnet-apac` — with a 400 ("…your request did not allow prompt caching"). The agent sends that call again without marks, logs `prompt caching refused for <model>`, and sends that route no marks again until the process restarts. The run's result is unchanged; the refusal costs one failed call per process.
- **How the marks reach Bedrock.** The control plane and runners call the LiteLLM gateway as an OpenAI-compatible client, and the marks travel in the request body; the gateway's Bedrock route turns them into Converse `cachePoint` blocks. A gateway alias that carries its own `cache_control_injection_points` stands down for a request that already carries marks, so the two never stack past the provider's limit of four breakpoints.
- **Not cached today: the `cx/*` (Sol) routes.** They reach OpenAI through a gateway that round-robins several accounts and reports no cached tokens; OpenAI-family routes get no marks.
- **How it is priced on Bedrock.** The AWS sync records each Claude model's cache-read and 5-minute cache-write rows (Claude Sonnet 5, global, ap-southeast-1: $0.20 and $2.50 per 1M against $2.00 input), so cached reads and writes book at AWS's own rates. A live run on 13 October 2026 read 60 % of its prompt from the cache and booked $0.1833 instead of $0.3020 uncached (−39 %). Off Bedrock, models.dev states the publisher's cached-read rate (and write rate where there is one), so a mapped model's cached reads book at the right rate wherever its endpoint reports them.
- **How to see it.** The cached share of a run's prompt tokens is recorded per model (`model_usage[model].cached_tokens`) and shown on the incident's Spend card; price cached reads with `cache_read_per_1m` (see *Cached prompt reads* above). A route that misses most of the time costs more than it saves: watch the cached share for a day after a new route starts serving investigations.
- **Turn it off.** `AIOPS_PROMPT_CACHE=false` for the platform, or `aiops_prompt_cache: false` for one workspace.

**Note for regulated customers.** The cached prefix — the system prompt with the workspace's knowledge base and estate, and the newest steps of the run — stays in the model provider's prompt cache for the cache lifetime (5 minutes on Bedrock, renewed on every hit). Bedrock and Anthropic isolate the cache per organisation: another account cannot read it, and a hit needs a byte-identical prefix. If keeping that prefix in the provider's cache for its lifetime is not acceptable for a workspace, set `aiops_prompt_cache: false` for it.

### The prompt diet

Off by default, because it changes what the model reads. With it on:

| Block | Off (today) | Diet |
|---|---|---|
| Skill catalog | every skill, one line each | the pillar's process skills in full, then whole lines up to 8,000 characters (the workspace's own runbooks before the shipped skills), then names only up to another 8,000, then a count with a `search_skills` tool |
| Estate | one block per connection, merged only where they overlap | one block for every connection, without the `grounded_by` lines and CloudFormation stack-id / logical-id tags, cut on a line at 100,000 characters with an `estate_lookup` tool behind the cut |
| Remediation routes | every enabled route | at most 4,000 characters, selectors cut at 200, then a count per mode |
| Blast radius | every impacted service | the nearest 20, then a count |
| Lakehouse | each table's row count | each table: `has rows` or `empty` |
| Planning rules | the full execution contract on every call | a summary on loop calls; the wrap-up still gets the full contract |

Budgets scale with the routed model's input window, as known to the model catalogue: full from a 400,000-token window up, proportionally below (an unknown window counts as 128,000). The tool-output history budget scales the same way.

Measure it before you turn it on: a Test Lab run takes `prompt_diet: true` or `false` (`POST /api/aiops/{tenant}/benchmarks/…`), a rerun keeps it, and `GET …/benchmarks/{run}/compare?baseline={run}` names each arm and says when the diet is the only difference between two runs of the same dataset, pinned model and grader. A run without a flag is recorded with the workspace setting it built with (`prompt_diet_resolved`), and two runs whose fingerprints match — the overflow guard dieted both — are not an A/B.

**The overflow guard.** With the diet off, a prompt that cannot fit the routed model's **known** input window — with room for the answer — gets the diet's budgets for that run, and the transcript records a routing note: `routing: prompt diet applied — the assembled prompt (~N tokens) leaves no room in <model>'s W-token input window`, naming the model whose window is the smaller one. A prompt that fits, or a model the catalogue does not know, changes nothing. It judges the models the run starts on: a Test Lab run's pinned model, and for a critical or production incident (whose loop runs on the reasoning tier) the reasoning model, not the standard one; a resumed run decides what its start decided. The window of an inference profile the catalogue does not list — the in-region alias's `apac.` profile — is read from the same model's other region profiles (200,000 for Claude 3.5 Sonnet v2), not from its base entry.

## Limits and gotchas

- **No hot reload.** These three settings are read once at process start. Nothing in the product changes them, and nothing displays them.
- **No overlap window means dropped runs.** The platform cannot hold a call while you rotate. If the gateway invalidates the old key immediately, in-flight calls fail; resumable runs retry up to their attempt allowance and then fail with their incident escalated.
- **Benchmarks key off `AIOPS_MODEL` alone.** The decision to run benchmarks through the real engine is made at startup from `AIOPS_MODEL`. A deployment configured **only** through a tier map — with `AIOPS_MODEL` left empty — routes real investigations correctly but still grades benchmarks with the mock runner. If you want a meaningful back-test, set `AIOPS_MODEL` as well as the tier map.
- **`AIOPS_API_BASE` is global.** It is applied to every call the platform makes, including calls whose model string names a different provider. Do not set it when your model string is `bedrock/…` or another non-OpenAI provider.
- **Model discovery is unauthenticated against your gateway's allow-list, and deliberately not SSRF-guarded.** `AIOPS_API_BASE` is an operator-level setting from the environment and never tenant input, so the discovery call is exempt from the guard that applies to tenant-supplied URLs. Point it only at an endpoint you control.
- **The gateway model list needs a permission.** Superadmin on `/ops/platform`; `pillars.manage` for a tenant admin on **AI models**. A tenant user without it sees a hint under the model pickers on **Agent behaviour**, the Test Lab (Live and Replay) and the Estate x-ray launch dialog instead of a list, and types ids under **Custom model id…** — that is a permission symptom, not a gateway failure.
- **litellm must be installed.** The shipped container image includes it. A custom install without the `agent` extra fails on the first real model call; the mock path never touches it.

## Troubleshooting

| Symptom | Likely cause | Do this |
|---|---|---|
| Dropdowns show only **Custom model id…**, page reads `no gateway configured` | `AIOPS_API_BASE` is empty in the running process | Set it and restart; confirm you edited the environment the process actually reads |
| Dropdowns empty, error names a connection or timeout failure | The control plane cannot reach the gateway, or the base URL is wrong | Curl `<AIOPS_API_BASE>/models` from inside the control-plane container with the same bearer |
| Dropdowns empty, error names a 401 or 403 | Wrong or revoked key | Rotate per the procedure above |
| Dropdowns empty but investigations work | Your provider serves no OpenAI-style `/models` list (Azure OpenAI, direct Bedrock) | Expected. Enter model ids under **Custom model id…** |
| Investigation card still reads `mock` after a restart | `AIOPS_MODEL` is empty and no tier map resolves, or the restarted process is not the one running jobs | Check the value in the process; in `remote` mode check the runner, not the control plane |
| Every run errors with a routing or provider error | The model string has no provider prefix | Prefix it, e.g. `openai/your-id` |
| Runs finish instantly with a root cause and no tool timeline | The model does not support tool calling | Use a tool-calling model; this is not recoverable by configuration |
| Root cause is a wall of prose with confidence `0.0` | The model or gateway ignored the structured-output request on the wrap-up call | Route the `reasoning` tier to a model that honours structured output |
| Cost reads `$0.0000` while tokens climb | The model id is on no card and absent from litellm's table, or a workspace endpoint model is not mapped yet (*Not priced yet*) | **Map** it to its publisher's model (or record a manual price); set the fallback per-million rates for the platform's caps |
| A tenant's cost cap never fires | Same cause — a cap cannot see zero | Same fix |
| **Needs attention** lists a model *not mapped to a publisher model* | No reviewed rule names it | **Map** it from the catalogue (suggestions first), or record its contract price with a note |
| **Needs attention** shows a change *waiting for your confirmation* | models.dev moved a rate beyond 2x (or a first mapping would) | Check the publisher's page, then **Confirm** — or leave it: the last price keeps applying |
| A Bedrock model reports *states no aws_region_name* | The gateway entry has no region | Add `aws_region_name` to that entry of the gateway config, then fetch again |
| **Fetch list prices** reports *Price list unreadable* | No outbound HTTPS to `pricing.us-east-1.amazonaws.com` | Open egress, or set `AIOPS_PRICE_SYNC_INTERVAL_S=0` and record manual prices |
| The models.dev line reads *the last read failed*, or *Needs attention* says *models.dev not read since …* | No outbound HTTPS to `models.dev` | Open egress, or set `AIOPS_MODELS_DEV_PRICES=false` and record manual prices; the last good catalogue keeps pricing meanwhile |
| A corrected price did not change last month's figures | Prices apply from their date; booked history moves only through the re-book | Record the price applied from when it was in force, then `POST /api/aiops/admin/rebook-costs` (dry run first) |
| Chat works, investigations and scans fail on authentication | `remote` mode: the key was rotated on the control plane only | Roll the runners; on the Docker driver, remove the container to force re-provisioning |
| An investigation sits at *The agent is thinking…* for minutes | The gateway is retrying a heavy model with backoff | Wait, or configure `AIOPS_FALLBACK_MODELS` so a flaky model fails over in seconds |

Changes made on **Platform defaults** are recorded in the cross-tenant audit log as `platform.model_tiers_default`. The environment settings in this chapter are not audited — they leave no trace inside the product, so record them in your own change management.

![The cross-tenant audit log: filter by tenant and by an exact action such as platform.model_tiers_default, which a tier-map change writes; the gateway environment settings leave no row.](/help/media/ops/ops-audit.png?v=0.1.132)


## Next steps

- Next: [Upgrade and release verification](/help/ops-upgrade-and-verify.md)
- [All Install and operate the platform pages](/help/install-and-operate-the-platform/llms.txt)
- [Documentation home](/help/)