Connecting the LLM gateway

Three deployment settings that give the platform a model, worked gateway configurations, how to verify them, the built-in mock model, where model prices come from, and rotating a key safely.

How-to · Updated 2026-10-17 · Describes release 0.1.164

What this is for

Every agent in this product — incident investigations, insights chat, estate x-ray, benchmarks — is one loop of tool call → tool result → next decision, driven by a large language model. The product ships with no model configured. Until a platform operator points it at a model endpoint, every one of those surfaces still works, still clicks through end to end, and still produces a report — from a deterministic mock model that invents nothing new and proves nothing about your estate.

This chapter is the one piece of configuration that turns the demo into a product. It is deployment-level work: three environment settings on the control plane (and on any runner processes), applied by whoever owns your deployment. There is no in-product screen that sets them, and no RBAC permission that grants them — an operator with shell or chart access to the deployment is the person who does this.

Once a model is configured, which model does which work is a separate, in-product decision made on Platform defaults and on each tenant's AI models page. That is covered in Platform configuration; this chapter covers getting a working endpoint underneath it.

Concepts

Gateway. Any HTTP endpoint that speaks the OpenAI chat-completions API. A litellm proxy, OpenRouter, an enterprise LLM gateway, a vLLM server, Ollama's compatibility endpoint — the platform does not care which, and has no special code for any of them.

Model string. A litellm model identifier such as openai/gpt-4o or bedrock/amazon.nova-pro-v1:0. The prefix before the first / selects the provider; the rest is the model id that provider understands.

Completion client. The only LLM dependency in the product is litellm, installed as the agent extra (the shipped container image installs it). Our agent loop is our own code; litellm is nothing but the multi-provider HTTP client underneath it.

Tier. A category of work — premium, reasoning, standard, light — that a model string is mapped to. Tiers are configured in the product; the settings in this chapter are what tiers resolve to when nothing more specific is set.

The three settings

Every setting in this product is read from the process environment with a AIOPS_ prefix. These are the real variable names:

Two related settings belong in the same change:

All of these are read once, at process start. Changing any of them requires restarting the process that reads them — there is no reload endpoint and no hot path that re-reads the environment.

Where you set them

Docker Compose

The compose file maps unprefixed names from your .env file into prefixed container environment:

environment:
  AIOPS_MODEL: ${AIOPS_MODEL:-}
  AIOPS_API_BASE: ${AIOPS_API_BASE:-}
  AIOPS_API_KEY: ${AIOPS_API_KEY:-}

So in .env you write AIOPS_MODEL=... without the prefix, because Docker Compose — not the application — reads that file for substitution. If you ever run the application directly against a .env file, the prefix is mandatory there: a bare AIOPS_MODEL= line is invisible to the app.

Helm chart

The chart splits them deliberately. The model string and the endpoint are non-secret and land in a ConfigMap; the key is a Secret:

config:
  AIOPS_MODEL: "openai/gpt-4o"
  AIOPS_API_BASE: "https://llm-gateway.example.com/v1"
  AIOPS_FALLBACK_MODELS: ""
secrets:
  AIOPS_API_KEY: ""

Set secrets.AIOPS_API_KEY with --set-string from your CI secret store, or point existingSecret at a Secret whose keys match these names. Anything the chart does not model — an extra provider variable, for instance — goes in extraEnv as a {name, value} entry.

CI pipeline

The shipped pipeline passes the key as a masked variable and the model and endpoint as plain config on the Helm upgrade. Keep the key in the deployment environment only; it does not belong in a values file that is committed.

A note on variables the app does not model

The application's settings object ignores environment variables it does not recognise, and litellm reads its own provider variables straight from the process environment. That is how provider-specific settings such as AZURE_API_VERSION and the standard AWS_* variables reach litellm even though this product never declares them.

How a model string routes

litellm resolves the provider from the prefix of the model string:

This is why the openai/ prefix matters. Gateways advertise their own bare ids — gpt-4o, claude-sonnet, internal/llama-70b. When the platform reads a gateway's model list it prefixes each id with openai/ so the value in the dropdown is directly usable as a model string. The prefixing is idempotent: an id that already starts with openai/ is left alone. If you type an id by hand anywhere in the product, you must add the prefix yourself.

Every call the platform makes carries a 300-second timeout and two litellm-level retries, with the OpenAI SDK's own slow backoff disabled — a flaky model fails over to AIOPS_FALLBACK_MODELS in seconds rather than sitting in per-attempt backoff for minutes.

Worked configurations

An OpenAI-compatible proxy

The common enterprise case: one gateway in front of several vendors, one key, one allow-list.

AIOPS_MODEL=openai/gpt-4o
AIOPS_API_BASE=https://llm-gateway.example.com/v1
AIOPS_API_KEY=sk-your-gateway-key
AIOPS_FALLBACK_MODELS=openai/gpt-4o-mini

The base URL must be the one whose /models and /chat/completions paths are valid — usually the one ending in /v1. This is the only configuration in which the gateway model dropdowns fill in, because model discovery is a GET against AIOPS_API_BASE + /models.

Azure OpenAI

Azure addresses a deployment name, not a model name, and needs an API version:

AIOPS_MODEL=azure/your-deployment-name
AIOPS_API_BASE=https://your-resource.openai.azure.com
AIOPS_API_KEY=your-azure-key
AZURE_API_VERSION=2024-10-21

AZURE_API_VERSION is read by litellm from the process environment; add it through extraEnv on the chart or the service environment block in compose. Azure OpenAI does not serve an OpenAI-style /models list at this base URL, so the model dropdowns stay empty and you enter ids under Custom model id…. That is cosmetic; routing works.

AWS Bedrock behind a gateway

Put Bedrock behind your gateway and it is an OpenAI-compatible proxy like any other — this is the configuration to prefer, because it keeps AWS credentials out of the platform entirely:

AIOPS_MODEL=openai/bedrock-claude-sonnet
AIOPS_API_BASE=https://llm-gateway.example.com/v1
AIOPS_API_KEY=sk-your-gateway-key

Calling Bedrock directly is also supported. Use a bedrock/ model string and give the process standard AWS credentials; leave AIOPS_API_BASE empty, because it would be applied to the Bedrock call as well.

AIOPS_MODEL=bedrock/amazon.nova-pro-v1:0
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION_NAME=ap-southeast-1

litellm looks for AWS_REGION_NAME first, then AWS_REGION. Set both to the same region: agent runners receive only AWS_REGION, and the platform refuses to price a model from the AWS Price List while the control plane and its runners would call different regions. Direct Bedrock has no /models list either, so the dropdowns stay empty.

Behind a gateway, state the region on each Bedrock entry of the gateway's config (litellm_params.aws_region_name): it is how the platform knows which region's AWS price applies — see Official prices: the AWS Price List.

A regulated workspace: an in-region Bedrock route

A workspace that must keep its model traffic inside a geography needs a gateway alias whose backing model is a regional inference profile, and a residency policy that refuses every other route. The shipped gateway configuration carries one such alias:

model_list:
  - model_name: bedrock/claude-3-5-sonnet-apac
    litellm_params:
      model: bedrock/apac.anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_region_name: ap-southeast-1

An apac. profile is served inside the APAC regions only: the entry point is the stated region, and Bedrock may fulfil the call in another APAC region, never outside the geography. It does not promise single-region processing. The platform also accepts a plain in-region model id when the effective region is known. Confirm current model access and profile availability in the account before publishing an alias; an entry that errors is worse in the tier dropdown than one that is absent.

1. Publish the alias through the normal reviewed change to the gateway configuration. Confirm from the gateway's /model/info that the entry reports its backing model and aws_region_name; an alias name alone is no residency evidence, and an entry that pins model_id is called at that id, so inspect that target too.
2. Fetch and review the alias's prices on Platform defaults → Model price map. Do not give an unresolved alias an invented price.
3. In the workspace, pin every tier the workload uses to the allowed route on AI models → Model tiers, and review any pillar override or fallback that could change the effective model.
4. On Guardrails & cost → Model residency, turn on Singapore regulated (MAS/PDPA) and read the saved policy. Under it, global profiles, other geographies, non-Bedrock routes and routes whose region cannot be placed are refused before a model call.
5. Run one bounded, authorized test and read the Model route recorded or Model denied by route policy audit row with its alias, backing model and region. A successful answer alone proves nothing about the route. Keep the policy and the routing evidence together.

This is evidence of the model route the platform used, not legal certification. The mock model makes no model call and proves no regional provider path. A Bedrock model that refuses prompt-cache markers is retried without them and that refusal is remembered for the process lifetime; the retry is not a reason to change region. Workspace guide: Models, tiers and routing → Set it up like the demo.

A self-hosted vLLM or Ollama endpoint

Both expose an OpenAI-compatible surface, so both are the proxy case with a local URL. vLLM's id is the --served-model-name you started it with:

AIOPS_MODEL=openai/your-served-model-name
AIOPS_API_BASE=http://vllm.internal.example.com:8000/v1
AIOPS_API_KEY=not-used-but-set-something

For Ollama, use its compatibility endpoint and the tag you pulled:

AIOPS_MODEL=openai/qwen2.5:14b
AIOPS_API_BASE=http://ollama.internal.example.com:11434/v1
AIOPS_API_KEY=ollama

Set a non-empty key even when the server ignores it; some clients reject an empty bearer. Both servers do publish /models, so the dropdowns fill in.

Before you commit to a self-hosted model, confirm it does tool calling. The agent loop hands the model a tool schema on every step and acts on the tool calls it returns. A model that cannot emit tool calls does not fail loudly — it answers in prose, the loop sees no tool call, and the run finishes immediately with a root cause and no evidence and no tool timeline. The same class of problem shows up at the final step: the wrap-up call asks for a structured JSON object, and if the model or gateway ignores that request the platform salvages what it can and otherwise records the raw text as the root cause with a confidence of 0.0.

Verify the gateway took effect

Do all four checks. Each proves a different layer, and the first three can pass while the fourth fails.

Requires superadmin for steps 1–3; a tenant member with report.view for step 4.

1. Restart the control plane after applying the settings, and confirm it came up.
2. Open /ops/platform (Platform defaults under Config). The four tier pickers should list your gateway's model ids, each prefixed openai/. This proves the endpoint and the key are correct — the list is fetched live from AIOPS_API_BASE + /models with AIOPS_API_KEY as a bearer. If the pickers offer only Custom model id… and the page reads "Gateway model list unavailable (…)", the message in brackets is the actual failure; no gateway configured means AIOPS_API_BASE is empty in the process you restarted.
3. Trigger one investigation in a tenant and open the incident. The investigation card's subtitle should name your model id, not mock. That proves routing.
4. On the same card, read the token and cost figures. Non-zero tokens prove a real call happened. A non-zero dollar figure additionally proves the model id is priceable — see the next-but-one section if it stays at $0.0000.
5. Open Chat in the tenant and ask something about your estate. A real answer varies with the question. The mock model always answers with the same sentence shape: "The system currently reports N incident(s) (source: sys_incidents)."

There is no page anywhere in the product that displays the configured AIOPS_MODEL, AIOPS_API_BASE or AIOPS_API_KEY. The dropdown filling in is the closest thing to a status indicator, and it is the check to teach your operators.

The mock model

When nothing resolves to a model string — no override, no tier map, no AIOPS_MODEL — the platform selects a deterministic scripted client instead of failing. It makes no network calls and needs no credentials.

It exists so that a fresh deployment is demonstrable and testable end to end before anyone provisions an LLM budget: correlation, the tool timeline, the write refusal, approvals, the root cause analysis block, the x-ray read-only denial and the artifact all render from real product code paths with a scripted brain behind them. It is also what makes the test suite hermetic.

How to recognise a mock run

Why every demo root cause analysis looks identical

Because the wrap-up is not generated — it is a fixed dictionary with your prompt's first 80 characters pasted into one field. Two different incidents produce two root causes that differ only in that fragment. If a customer says "the analysis is always the same", that is not a model quality problem; that deployment has no model configured.

What the mock does not prove

• Nothing about accuracy. The scripted responder cannot match ground truth, so the platform deliberately does not run benchmarks through the real engine when no model is configured — it uses a ground-truth echo runner and labels the result honestly. An 80% pass rate from a mock deployment measures the pipeline, not the agents.
• Estate x-ray spend is fictional. The mock x-ray client reports a flat $0.50 per call, so a mock scan accumulates a plausible-looking cost and can even hit its budget. Those dollars were never spent.
• The gateway itself is untested. A mock run passing tells you nothing about credentials, network reachability or tool-calling support.

The default tier map and what a tenant inherits

AIOPS_MODEL is the floor, not the routing policy. Work is routed by tier — premium for x-ray scans, reasoning for the root cause wrap-up, standard for the investigation loop and chat, light for bulk work — and each tier maps to a model string.

A tenant that has never configured its own tier map inherits the platform default map you set on Platform defaults. A tenant that sets even one tier owns the whole map: the two are never merged, which keeps a tenant's spend predictable. A missing tier resolves to the nearest configured tier above it, then below, so setting only standard gives a working four-tier platform.

Two consequences worth knowing before you configure anything:

• The tier map is the routing policy; AIOPS_MODEL is what remains when no map applies. With one model and no map, every tier is that model and mid-run tier escalation has nowhere to go — the escalation ladder needs at least two distinct configured models to have a step to take.
• In a remote deployment the runners follow the same map, from 0.1.117. An api-mode runner receives the control plane's effective map with every job and a db-mode runner re-reads it each time it polls, so a tenant with no map of its own runs on the platform default there too, not on the runner's AIOPS_MODEL. A runner image older than 0.1.117 never sees the map and runs such a tenant on its own AIOPS_MODEL_TIERS_DEFAULT, else on AIOPS_MODEL.

The full resolution order, how runners receive the default map and its environment-variable form are in Platform configuration — do not maintain a second copy of that chain.

Cost, tokens and the pricing map

Per call, the platform records prompt tokens, completion tokens, cached reads, cache writes and a dollar cost, and adds them to the run's row. Spend is booked in three places:

Every booked figure also records where its price came from — price_source on each model's line of the run (model_usage[model]), and price_ref, the card entry that priced it:

A run priced from an estimate still counts against budgets and the per-run cap — an estimate is better than $0 for a cap — but every cost surface says which part of a figure is an estimate and why: Billing in this console, a workspace's Usage & billing, an incident's Spend card and the report. A run booked before price sources were recorded is read as an estimate when its model's line carries the old fallback flag and money.

The model library's table is the estimate, and it is pinned. litellm prices a call from a community-maintained table. Unpinned, it downloads that table from GitHub every time a process starts, so one release booked different estimates depending on which pod reached GitHub — measured on 13 October 2026: Claude Sonnet 5.5 at $0 from the bundled copy, an apac. Claude Sonnet 5 at the global $2.00 instead of the regional $2.20, an apac. Nova Pro at $0.84 per 1M where AWS charges $1.08 from Singapore. Both images set LITELLM_LOCAL_MODEL_COST_MAP=True: an estimate is the table the release shipped, labelled with the litellm version that produced it. A call through a LiteLLM gateway is the exception: litellm books the cost the gateway returns in its x-litellm-response-cost header, computed from the gateway image's own table (which moves with that image and what it downloaded), so its price_ref reads gateway x-litellm-response-cost, never this release's version. The platform never writes the card into that table: every client books from the card itself, and a registration would outlive the card — a removed price kept booking until a restart, and another gateway's model sharing a bare id with a card entry booked the card's rate as the library's estimate. A gateway id litellm does not know at all prices at $0 — see Set the fallback rates.

Official prices: the AWS Price List

For a model the gateway serves from AWS Bedrock, the platform reads the price AWS publishes — the public Price List files, no credentials, no AWS account:

1. Alias → backing model and region, from the gateway's own GET /model/info: litellm_params.model and litellm_params.aws_region_name. An entry that pins litellm_params.model_id is called at that id, so that is the target priced (and judged by a model route policy): a global. profile pinned there bills the global rows; an application inference profile or provisioned-throughput ARN hides the model and stays unpriced from AWS. The gateway entry must state aws_region_name — the platform never assumes the region the gateway pod happens to run in. With no gateway (AIOPS_API_BASE empty) and a bedrock/… model, the region is the control plane's AWS_REGION; it is refused when AWS_REGION_NAME says otherwise, because runners only receive AWS_REGION.
2. Backing model → price-list product, through a reviewed table (BEDROCK_PRODUCTS in app/aiops/price_sources.py) keyed by the exact Bedrock base id and the price list's exact model name. Claude Sonnet 5 and Claude Sonnet 5.5 share every AWS usage type and differ in the cache-read rate ($0.20 and $0.10 per 1M), so only the exact name may select rows. A model the table does not name stays unpriced from AWS and is reported; it is never matched by a similar name.
3. Inference profile → rows. A global. profile takes the global rows; a geographic profile (apac., us., eu., …) or an in-region model id takes the regional rows of the source region — the region the gateway calls from. AmazonBedrockFoundationModels prices Anthropic and the other Marketplace-billed models; AmazonBedrock prices the Amazon models (Nova, listed per 1K tokens and converted to per 1M).
4. Rows → rates. Only the standard on-demand text rows: input, output, cache read and the 5-minute cache write. Batch, flex, priority, latency-optimized, long-context, 1-hour cache-write, custom-model and provisioned-throughput rows never match. A rate the list does not state stays absent (Claude 3.5 Sonnet v2 has no cache rows in ap-southeast-1). Two different prices for the same kind price nothing. A $0 input or output price is never applied unattended.

What does not resolve stays unpriced and is listed with its reason under Needs attention on Platform defaults → Model price map: the gateway entry for … states no aws_region_name, no reviewed AWS mapping for …, the gateway balances … across deployments with different models or regions, no regional on-demand input and output price … (a model sold only through the global profile in that region), or the platform gateway does not route …. An alias the gateway serves from Bedrock is never priced from models.dev, even when BEDROCK_PRODUCTS lacks its model: add the reviewed row, or record a manual price citing the AWS list. Azure OpenAI has no price source yet; enter its price manually.

Fetch list prices runs the read now (POST /api/aiops/admin/model-prices/sync, superadmin with an admin-scoped token) — both parts: the AWS Price List here and models.dev below. The control plane repeats it daily for the models in use — booked in the last 30 days, or configured as a tier, AIOPS_MODEL or a fallback — on the scheduler leader only: AIOPS_PRICE_SYNC_INTERVAL_S (default 86400; a failed attempt retries after an hour; 0 turns both parts off for an install without outbound HTTPS to pricing.us-east-1.amazonaws.com and models.dev). Per Bedrock model the result reads Added, Changed, Unchanged, Kept the manual price, Kept removed, Skipped ($0 on the list), Price list unreadable, Not mapped or No official source.

• A new or changed price is recorded from that moment, with one platform.model_price audit row; an unchanged one writes nothing. A price whose numbers match but whose product changed — an alias re-pointed from Claude Sonnet 5 to 5.5 — is a new entry.
• A manual price outranks the list price: the sync keeps it, and keeps a model you removed, unless you name the model — Replace with the list price beside a kept manual price.
• A download that fails changes nothing: the card keeps its last good prices and the result says the price list could not be read. A gateway whose /model/info cannot be read (restarting, say) fails the read the same way — every Bedrock alias is reported Not mapped — and it is retried within the hour, not the next day.
• Nothing is re-priced: booked history moves only through the re-book.

After you change the gateway's model list, press Fetch list prices: until the next read, an alias re-pointed to another model books at the old model's price.

Publisher list prices: models.dev

No model publisher offers a price API. For every model not served through Bedrock, the platform reads models.dev (https://models.dev/api.json), an open, MIT-licensed community dataset of model prices — and only each publisher's own entry in it: OpenAI, Anthropic, Google (Gemini API), DeepSeek, Z.ai, Mistral, xAI, Moonshot AI, Alibaba Cloud (international) and Meta. Resellers and hosts (OpenRouter, Azure, Amazon Bedrock, Vertex), subscription plans listed at $0 (*-coding-plan) and the CNY price lists are never read — nor another publisher's model a publisher's own block resells (models.dev lists GLM, DeepSeek and Kimi models under Alibaba and Mistral too; the model's canonical publisher decides, so only Z.ai's, DeepSeek's and Moonshot's own entries price them). On 13 October 2026 it matched the publisher pages on 28 of 30 models; the two misses were DeepSeek's off-peak-only rows (see gaps below).

• Catalogue. Each priced publisher model becomes a dated entry under a reserved key — models.dev/openai/gpt-5.6-sol — append-only like every card entry: a change is a new entry from the moment it is read, never an edit. The entries never ship as card keys; Publishers and workspace endpoints → Publisher catalogue lists them with their history, gaps and the publisher's pricing page.
• Mappings. An alias reaches its publisher's price through a dated mapping: from this date, alias X is publisher model P. A mapping is resolved when a price is read, so one publisher change re-prices every alias bound to it from its date. Every spelling runs book under is mapped on its own (openai/glm/glm-5.3 and glm/glm-5.3).
• Only the reviewed router table maps unattended: cx/ → OpenAI (a trailing -review is the same model on another quota), glm/ → Z.ai, ds/ → DeepSeek, cc/ → Anthropic, applied to the backing the gateway's /model/info reports (else to the alias). cx/gpt-5.3-codex-spark has no publisher price — OpenAI publishes none and models.dev's copied figure is refused. A mapping the sync recorded follows a re-pointed backing: to another model the table names, it is re-mapped from that read; to one with no publisher price behind the table — outside it, not listed by models.dev yet, no publisher price, served from Bedrock, or balanced over deployments that serve different models — it ends from that read and the alias is listed as not mapped (it never keeps booking the old model's price). A mapping an operator recorded is never changed, only reported when the table disagrees. While the gateway's /model/info cannot be read, the table changes nothing.
• Everything else is mapped by you, from the catalogue: Map beside a model under Needs attention offers suggestions first — high confidence: the router table, the gateway's own model_info.base_model, a native backing (anthropic/…, gemini/…); low confidence: the model the endpoint said served the calls (recorded on every usage line — text a customer's endpoint or agent reports) and a match by name alone — then a search grouped by publisher. The dialog opens on a high-confidence suggestion only; a low-confidence one is listed for you to check and pick. Map all suggested applies the high-confidence ones in one confirmation; a low-confidence one is never applied in bulk. A mapping may not start before its model's first catalogue entry; a run before that is priced by a manual entry, if at all. POST /api/aiops/admin/model-prices/mappings records the same rows (a note on each; if_current refuses a stale page with 409).
• Held changes. A change that moves any rate beyond 2× either way — a unit slip, a batch rate published as the standard one (models.dev showed GPT-5.6 Sol at its batch rate for days in August 2026) — waits under Needs attention until you press Confirm (confirm on the sync). So does a first mapping that would move an alias's current price that far, or off a $0. For a publisher model, a cache rate that appears or disappears counts by what it does to the bill (absent = billed at the input rate): models.dev dropping GPT-5.6 Sol's cached-read rate would bill cached tokens ten times higher, so it waits too. Confirm applies the change it showed you: if models.dev publishes another value before you press it, that value waits again with its own numbers. A held change stays listed through a failed read.
• Gaps are flagged, never filled in. DeepSeek bills twice its listed rate at peak hours; models.dev lists the off-peak rate, so DeepSeek lines book as estimate. A reasoning rate that differs from the output rate books as estimate. A long-context tier (OpenAI above 272K prompt tokens, Gemini Pro and xAI above 200K, Claude Haiku 5.5 above 100K, three Alibaba models above 32K) is kept as a threshold: a call over it books at the base rate as estimate. Dated prices models.dev cannot state — GPT-5.6 Sol's promotional price "at least through November 21, 2026", Mistral Large 4's sale price, Gemini 3.8 Flash's 1 January 2027 rise — are a reviewed watch list: from the review date Needs attention asks you to check the publisher's page. Batch, flex, priority and regional uplifts are never requested by the platform; the standard global rate is the one it books.
• Safeguards, as for the AWS part. One fixed URL, no redirects, TLS, 16 MB and 40 seconds, a conditional read (If-None-Match), a shape check; any failure keeps the last good catalogue and every mapping, and the hourly retry follows. A $0 price is never applied, a manual price is never overwritten, nothing is re-priced. AIOPS_MODELS_DEV_PRICES=false turns this part off (the AWS part keeps running).
• What a customer sees: OpenAI list price · gpt-5.6-sol · models.dev, read 14 Oct 2026, and on the statement one sentence naming the sources that priced the month — models.dev and its licence when it priced any. A DeepSeek or long-context estimate is said as a list price that states part of the bill, never as a missing price.
• Back-dating is the same human act as for an AWS price: models.dev follows a publisher change within days, so when the gap matters record a manual entry with the publisher's rates from the publisher's date (its note citing the publisher's page and the models.dev revision), press Replace with the list price so the mapping rules again from now, then re-book (dry run first).

The card keeps its history

Every price is an entry with its source, its note, who set it and when, and the date it applies from. A save never rewrites an entry: a corrected rate is a new entry, and the database refuses an edit to an old one. Each entry says this rate from this date on, so the price in force for a model at any moment is the entry recorded last among those applied by then — a later save outranks an earlier one from its own date on, even one dated after it — with the deployment's own map underneath for models no entry covers. The deployment's map is rates only: a source written into an AIOPS_MODEL_PRICES entry is ignored, and its rates read source not recorded until someone records them. Each run is priced at the entry in force when it started — the cost split on every surface and the re-book alike — so a price change never re-prices the past on its own.

• A manual price needs a source note — the contract or the pricing page it comes from. The editor refuses a changed, new or removed price without one (PUT /api/aiops/admin/model-prices answers 400 naming the models), and a price that does not state both its input and output rate (0 states a free model). Record source under Needs attention records the rate that prices the model now, unchanged, with your note — under the model id runs book with, wherever the deployment's map keys it (for example without the platform's openai/ prefix).
• Apply from dates the edited prices (today by default, never in the future). The price applies from that date on, the present included, over every price recorded before it — an official AWS entry dated later too, which the daily sync then keeps as a manual price. A price applied from today leaves the runs that already happened as they were booked. To correct the past, apply it from when the rate was really in force — the editor suggests the edited models' first booked run — then re-book. To correct only a past period, save that period's rate from its start, then the rate that followed from its own date.
• A save carries the card's version (version from GET). When the card changed after the page loaded — the daily sync recorded a price, another administrator saved — the save is refused (409) and the page reloads the card: make the edit again. A full map saved over a newer card would otherwise remove the prices it does not hold, or write its stale rates over them as manual prices the sync then keeps.
• A model left out of the map is removed from now (an entry too, with the note saying why). An empty map is refused.
• Back-dating an official price is a human act. The daily sync starts an official price now: runs booked before it stay estimates until a superadmin records the price that applied then — a manual entry applied from when the alias began routing to that model, with a note citing the AWS publication or the publisher's page — and re-books. Like any save, that entry rules from its date on, the present included, so the sync would keep it as a manual price: press Fetch list prices, then Replace with the list price beside it, and the list price rules from now again while the manual entry covers the past. The platform keeps no per-run record of which model an alias meant on a past date, so it does not back-date on its own.
• A mapped alias in the editor shows its publisher's rates; editing them records a manual entry for the alias that outranks the mapping from its date. A row for a catalogue key (models.dev/openai/gpt-5.6-sol) records an override of that publisher model for every alias mapped to it.
• Only rates, the source, the entry id, the date and — for a models.dev price — the long-context threshold and gap codes travel to agent runners (customer-run private agents included). The note, the page, the revision and who set it stay on the control plane; a workspace sees the kind of price, its date and, for an AWS price, the region and publication date, for a models.dev price the publisher, model, read date and pricing page — never your note.

Set the fallback rates

The platform applies a per-token estimate whenever nothing priced a call that used tokens — no card entry and no litellm price:

AIOPS_PRICE_PER_1M_INPUT_USD=3
AIOPS_PRICE_PER_1M_OUTPUT_USD=15

Use the rates your gateway actually bills you. They apply to every unpriced model in the deployment, so on a multi-model gateway choose the rates of the model doing the bulk of the work. Calls priced this way are labelled flat-estimate and shown as estimates; they keep the caps working, they are not an invoice. Left at 0 the feature is off and an unknown id books $0.

Cached prompt reads — price them explicitly

An investigation replays its growing transcript on each step. The gateway may report cached prefix reads within its prompt tokens; their share and price depend on the gateway and workload. The platform records reported cached reads per model (model_usage[model].cached_tokens), and the incident Spend card shows their share. Use the rate your gateway actually bills rather than assuming a vendor ratio.

Their booked price depends on cache_read_per_1m on the card. This example is illustrative; enter your gateway's rates:

"openai/gpt-5": { "input_per_1m": 1.25, "output_per_1m": 10, "cache_read_per_1m": 0.125 }

• cache_read_per_1m present → cached reads book at that rate.
• cache_read_per_1m absent → cached reads book at the full input rate. This is deliberate: a missing rate over-counts, and over-counting is the safe failure mode for a billing number. Confirm any correction against your gateway's rate card and invoice; the platform does not assume a discount.

A missing rate is a visible signal, not a silent over-count. Every surface that shows money — GET /{tenant}/spend, an investigation, an incident's Spend card, the statement — carries a cost_breakdown: the same dollars split into fresh input, cached reads, cache writes and output, at the rates that applied when each run ran, per model, with each model's price source and these lists: unpriced (models no entry priced — their booked figure passes through undecomposed), cache_rate_missing (priced, cached reads, no cache_read_per_1m — the tenant is told the figure is an over-estimate), cache_write_rate_missing (cache writes billed at the input rate) and estimated (booked money that came from an estimate). The card's models with no cache-read rate show as cache read = input on Prices in force.

A rate entered from today clears the banner for runs from today on; the runs already booked keep their split and their figure. A rate applied from an earlier date re-prices the split of the runs it covers on the next read, while the booked cost_usd on each run does not move until you re-book: the payload then comes back rebook_needed: true and the tenant's page says what was booked, what the rates now recorded for that time put the same tokens at, and that re-pricing history brings them together. rebook_needed is also set when the money agrees but a run booked as an estimate is now covered by an official or contract price — the re-book only re-labels it (an estimate the price itself carries, a models.dev gap or a long prompt, stays one).

The fields are accepted wherever a price is entered: the platform card (Platform defaults → Model price map, PUT /api/aiops/admin/model-prices, or the AIOPS_MODEL_PRICES environment variable) and a workspace endpoint model's own rates (Map → Own rates). models.dev states the cached-read rate (and the write rate where the publisher prices one), so a mapped model gets it without an entry; a 0 cache rate there is dropped (absent = input rate). DB-free agent runners receive the platform card from the control plane with every job — a workspace gateway reaches API-mode agents only when its owner explicitly shares it for agents, and then with its own card (the operator's mappings over the catalogue). Its key is resolved through a workspace-scoped virtual secret reference, never returned in the configuration payload, and a daily catalogue change never rotates that reference.

Workspace endpoints (bring your own LLM)

For a model on a workspace's own endpoint, the billing price is the publisher's list price of the model it really is — never what the endpoint states: not a LiteLLM proxy's x-litellm-response-cost (their $0, or their price after their own discounts), not an OpenRouter-shaped /models price, not the rate card the workspace typed.

• Mapped models book their publisher's list price (models-dev) or the rates you recorded (manual); No token charge is your 0 / 0 with a written reason (self-hosted open weights, included by agreement, internal test).
• Unmapped models book $0 with no source: every surface prints Not priced yet, the ops invoice shows the tokens under Not priced — resolve before invoicing, and Needs attention counts them. The flat fallback never applies to a workspace endpoint, so the per-run cost cap cannot see an unmapped model's calls until you map it; steps and the context budget still bound a run.
• Who maps: the platform operator only — Publishers and workspace endpoints → Workspace endpoints lists each workspace's models with their runs, tokens, money and the models the endpoint said it served, with Map, Own rates and No token charge. A lab box uses its parent's mappings. Each mapping lands a platform.model_mapping row in that workspace's audit log ("Model price set by the platform operator"), without your note.
• The old rate card a workspace typed keeps pricing exactly the runs before the upgrade (byo_cards_retired_at, stamped once at control-plane startup); a card saved later prices nothing. PUT /{tenant}/llm-endpoint still accepts prices and stores them, and its answer says the platform sets the prices; GET lists each endpoint model with its price status.
• After mapping, Re-price history (dry run) on the workspace shows what moves — the workspace's runs, those of every lab box on its endpoint and the Test Lab's, the runs the group counts; re-price it from there or with the API below (include_lab_boxes, include_test).
• Only the endpoint's own lines are listed there: the runs a workspace made on the platform gateway before it switched (or beside it) stay under the platform's models.

Cache writes

Writing a prompt into the provider's cache is billed too: Anthropic models on Bedrock charge about 1.25 times the input rate for it (Claude Sonnet 5: $2.50 per 1M against $2.00 input), Amazon Nova charges nothing. The platform records cache-write tokens per model (model_usage[model].cache_write_tokens; they are part of the prompt tokens, so fresh input is prompt − cached reads − cache writes) and books them at cache_write_per_1m. With no write rate on the card they book at the input rate and the split flags cache_write_rate_missing, because the direction of the error depends on the provider. The AWS sync records the 5-minute write rate; a 1-hour cache write — the platform never asks for one — books at the 5-minute rate and marks the call an estimate.

Re-price booked history

Changing the card does not move what was already booked. POST /api/aiops/admin/rebook-costs re-prices history deterministically from the recorded per-model tokens, each run at the card entry in force when it started: {"tenant_id"?, "since"?, "until"?, "models"?, "include_test"?, "include_lab_boxes"?, "dry_run": true} — models moves only those model lines (every other line keeps its value and label), include_test also takes the Test Lab's runs (skipped otherwise; most of a lab box's spend is there), both recorded in the audit row; include_lab_boxes (with tenant_id) also takes every lab box that runs on that workspace's own endpoint — reports count, changed (runs whose money moved), relabeled (runs where only the price source moved — the first re-book after price history existed stamps unattributed on every run the deployment's map priced; that is bookkeeping, not money), booked_usd / rebooked_usd and a sample with each model's from → to source; dry_run: false writes the new cost_usd (run total and per-model split, with the price source) and lands one billing.rebooked audit row per tenant with both counts. Run the dry run first and read the sample. Models no entry priced at a run's start keep their booked value and label, benchmark runs (unless include_test) and runs still in flight are left alone. A workspace on its own endpoint re-prices through its own view — your mappings for its models (prices_source: "byo" per workspace) — while the lines its retired rate card priced keep their value and label. In every other workspace, a model line whose recorded provenance says a workspace endpoint booked it — byo: true, byo-card, or the library's estimate booked on a workspace gateway — keeps its value and label: a workspace that switched bring-your-own off keeps what its endpoint cost. Three numbers are frozen copies and do not move: a test-lab run's per-case metrics, an x-ray scan's cost_usd (summed from its phase runs once, at finish) and a validation recheck's cost_usd (no per-model split) — re-read the investigation rows for the corrected figure. A month the data-retention sweep has sealed is frozen too: its statement is read from the sealed row (Platform defaults → Data retention default), so re-booking changes neither the invoice nor the workspace statement for that month. Needs an admin-scoped token, like every /admin route.

Reconcile the tokens with AWS

The money is only as good as the tokens. For Bedrock models, compare what the platform booked with what AWS counted — read-only, from the control-plane pod so it uses the platform's own AWS identity:

kubectl -n aiops exec deploy/aiops -- python -m app.aiops.price_sources reconcile --days 7
kubectl -n aiops exec deploy/aiops -- python -m app.aiops.price_sources reconcile --since 2026-10-01 --until 2026-10-08 --json

Per Bedrock model id and source region (global.anthropic.claude-sonnet-5 @ ap-southeast-1) it prints the platform's fresh input, cache reads, cache writes and output against CloudWatch AWS/Bedrock InputTokenCount, CacheReadInputTokenCount, CacheWriteInputTokenCount and OutputTokenCount (dimension ModelId), the drift and a verdict: match (within 2%), aws-higher (other callers share the account or region — expected on a shared gateway) or ours-higher (the platform booked tokens AWS did not count: investigate). Models that are not Bedrock, or have no region, are listed apart. It needs cloudwatch:GetMetricData on the control plane's role; without it the command names the missing permission and exits 2. Nothing is written.

On a customer's own AWS account (BYOC), reconcile the money in Cost Explorer: filter Service to the price list's service name (for example Claude Sonnet 5 (Amazon Bedrock Edition), billed through AWS Marketplace, or Amazon Bedrock for Nova), group by Usage type, and compare the usage types each AWS price cites (shown on Prices in force) with Usage & billing → By model for the same UTC month. Quantity × the entry's rate is the list cost; a difference comes from other workloads in the account, a private price or credits (record a contract price as a manual entry — it outranks the list), or tax.

Rotate a key without breaking in-flight runs

A key rotation is a process restart. What matters is what happens to work that is mid-flight when the process goes down.

Embedded mode

The control plane runs agent jobs itself. On restart:

• Investigations from alerts, schedules and manual triggers that were running are re-queued and resumed from their committed transcript — the transcript is written at every step boundary, so a resume replays at most the one interrupted step. Each resume costs an attempt; the default allowance is three, after which the run fails and its incident is escalated.
• Estate x-ray scans in mission mode resume the same way, rehydrating their ledger.
• Chat turns, benchmark runs and OpenAI-facade calls are not resumable and are failed — the caller re-asks.
• Runs sitting in paused_approval are untouched and resume when a human decides.

Procedure. Requires deployment access; no product permission applies.

1. Mint the new key on the gateway and confirm the old key is still valid — an overlap window is what makes this safe.
2. Update AIOPS_API_KEY in the ConfigMap, Secret or .env that feeds the control plane.
3. Restart the control plane.
4. Open /ops/platform and confirm the model dropdowns still populate. That is the new key succeeding against the gateway.
5. Trigger one investigation and confirm it completes with your model id on the card.
6. Revoke the old key at the gateway.

You should see previously-running investigations continue and finish rather than appear as failures; the incident feed shows them advancing again within a sweep interval.

Remote mode and per-tenant runners

With AIOPS_EXEC_MODE=remote the control plane only enqueues work; per-tenant runner processes execute it. Restarting the control plane does not disturb a running job, which is the point of the topology — but it also means the control plane's new key does not reach the agents. The runners hold their own copy.

How the new key reaches a runner depends on how runners are provisioned:

In all three cases, an interrupted runner job is recovered by the same lease mechanism as an embedded restart: another runner steals the expired lease and resumes from the committed transcript.

Sequence for a remote deployment:

1. Confirm the overlap window at the gateway.
2. Update the key in the control-plane configuration and in the runner configuration.
3. Roll the control plane.
4. Roll or recreate the runners, one tenant at a time if you want to bound the blast radius.
5. Verify per tenant: trigger an investigation, confirm it completes on your model id.
6. Revoke the old key.

Update the gateway configuration used by private agents as well. A successful control-plane request does not prove that a remote investigation, scan or conversation can authenticate — hosted chat may succeed while runner investigations fail, and private-bound workspace chat runs on private agents. A hosted chat result does not validate a private agent’s model credential. Verify a real task through the relevant agent after rotation.

Prompt caching and the prompt diet

Two platform defaults decide what one agent call sends. Both are environment settings on the control plane, read once at process start like the rest of this chapter, and both have a per-workspace override in the tenant's settings (PATCH /api/aiops/{tenant} with {"settings": {...}}, a JSON boolean, or null to return to the platform default). Remote runners follow the control plane, not their own environment: the control plane resolves both values into the configuration it ships with every job (dbMode: api), and copies both variables into the environment of every runner it provisions (dbMode: db, which builds from the database). A runner you run yourself — the compose agents profile — needs them set in its own environment too. An agent image older than 0.1.117 ignores them and sends the prompt it always sent.

Prompt caching

An investigation sends its whole conversation on every step: the system prompt (persona, the workspace's knowledge, the estate, the planning rules, the skill catalog), the tool list and every earlier step. With caching on, each loop call to a Claude route marks two places with cache_control: {"type": "ephemeral"}: the system message, which caches the tool list and the system prompt, and the newest message of the conversation. The next call reads everything up to that mark from the provider's cache at the cache-read rate (0.1× input on Claude Sonnet 5) and writes only what is new, at the cache-write rate (1.25× input).

• Which calls. Only loop calls that another call with the same prefix follows. The wrap-up, the best-of-2 arbiter, the self-check, the last possible loop step and any run of at most two steps (the correlation sweep, storm triage) send no marks: a marked call that nobody reads again is a 1.25× write for nothing.
• Which routes. Claude is judged on the model id the platform gateway routes an alias to — its /model/info, the same map a model route policy reads, refreshed on a timer — never on the alias name. A native route (bedrock/…anthropic.claude…, anthropic/…) is its own id. Every other route sends no marks: an OpenAI-compatible endpoint rejects request keys it does not know. That includes a tenant's own gateway (the platform never asks it for /model/info) and a gateway with no /model/info. AIOPS_FALLBACK_MODELS counts too: a failed call is re-sent to every fallback with the same body, so the marks go only when every fallback is a Claude route as well.
• A provider that refuses the marks. Bedrock answers a request with cache points for a model whose prompt caching never left preview — Claude 3.5 Sonnet v2, the model behind the in-region alias bedrock/claude-3-5-sonnet-apac — with a 400 ("…your request did not allow prompt caching"). The agent sends that call again without marks, logs prompt caching refused for <model>, and sends that route no marks again until the process restarts. The run's result is unchanged; the refusal costs one failed call per process.
• How the marks reach Bedrock. The control plane and runners call the LiteLLM gateway as an OpenAI-compatible client, and the marks travel in the request body; the gateway's Bedrock route turns them into Converse cachePoint blocks. A gateway alias that carries its own cache_control_injection_points stands down for a request that already carries marks, so the two never stack past the provider's limit of four breakpoints.
• Not cached today: the cx/* (Sol) routes. They reach OpenAI through a gateway that round-robins several accounts and reports no cached tokens; OpenAI-family routes get no marks.
• How it is priced on Bedrock. The AWS sync records each Claude model's cache-read and 5-minute cache-write rows (Claude Sonnet 5, global, ap-southeast-1: $0.20 and $2.50 per 1M against $2.00 input), so cached reads and writes book at AWS's own rates. A live run on 13 October 2026 read 60 % of its prompt from the cache and booked $0.1833 instead of $0.3020 uncached (−39 %). Off Bedrock, models.dev states the publisher's cached-read rate (and write rate where there is one), so a mapped model's cached reads book at the right rate wherever its endpoint reports them.
• How to see it. The cached share of a run's prompt tokens is recorded per model (model_usage[model].cached_tokens) and shown on the incident's Spend card; price cached reads with cache_read_per_1m (see Cached prompt reads above). A route that misses most of the time costs more than it saves: watch the cached share for a day after a new route starts serving investigations.
• Turn it off. AIOPS_PROMPT_CACHE=false for the platform, or aiops_prompt_cache: false for one workspace.

Note for regulated customers. The cached prefix — the system prompt with the workspace's knowledge base and estate, and the newest steps of the run — stays in the model provider's prompt cache for the cache lifetime (5 minutes on Bedrock, renewed on every hit). Bedrock and Anthropic isolate the cache per organisation: another account cannot read it, and a hit needs a byte-identical prefix. If keeping that prefix in the provider's cache for its lifetime is not acceptable for a workspace, set aiops_prompt_cache: false for it.

The prompt diet

Off by default, because it changes what the model reads. With it on:

Budgets scale with the routed model's input window, as known to the model catalogue: full from a 400,000-token window up, proportionally below (an unknown window counts as 128,000). The tool-output history budget scales the same way.

Measure it before you turn it on: a Test Lab run takes prompt_diet: true or false (POST /api/aiops/{tenant}/benchmarks/…), a rerun keeps it, and GET …/benchmarks/{run}/compare?baseline={run} names each arm and says when the diet is the only difference between two runs of the same dataset, pinned model and grader. A run without a flag is recorded with the workspace setting it built with (prompt_diet_resolved), and two runs whose fingerprints match — the overflow guard dieted both — are not an A/B.

The overflow guard. With the diet off, a prompt that cannot fit the routed model's known input window — with room for the answer — gets the diet's budgets for that run, and the transcript records a routing note: routing: prompt diet applied — the assembled prompt (~N tokens) leaves no room in <model>'s W-token input window, naming the model whose window is the smaller one. A prompt that fits, or a model the catalogue does not know, changes nothing. It judges the models the run starts on: a Test Lab run's pinned model, and for a critical or production incident (whose loop runs on the reasoning tier) the reasoning model, not the standard one; a resumed run decides what its start decided. The window of an inference profile the catalogue does not list — the in-region alias's apac. profile — is read from the same model's other region profiles (200,000 for Claude 3.5 Sonnet v2), not from its base entry.

Limits and gotchas

• No hot reload. These three settings are read once at process start. Nothing in the product changes them, and nothing displays them.
• No overlap window means dropped runs. The platform cannot hold a call while you rotate. If the gateway invalidates the old key immediately, in-flight calls fail; resumable runs retry up to their attempt allowance and then fail with their incident escalated.
• Benchmarks key off AIOPS_MODEL alone. The decision to run benchmarks through the real engine is made at startup from AIOPS_MODEL. A deployment configured only through a tier map — with AIOPS_MODEL left empty — routes real investigations correctly but still grades benchmarks with the mock runner. If you want a meaningful back-test, set AIOPS_MODEL as well as the tier map.
• AIOPS_API_BASE is global. It is applied to every call the platform makes, including calls whose model string names a different provider. Do not set it when your model string is bedrock/… or another non-OpenAI provider.
• Model discovery is unauthenticated against your gateway's allow-list, and deliberately not SSRF-guarded. AIOPS_API_BASE is an operator-level setting from the environment and never tenant input, so the discovery call is exempt from the guard that applies to tenant-supplied URLs. Point it only at an endpoint you control.
• The gateway model list needs a permission. Superadmin on /ops/platform; pillars.manage for a tenant admin on AI models. A tenant user without it sees a hint under the model pickers on Agent behaviour, the Test Lab (Live and Replay) and the Estate x-ray launch dialog instead of a list, and types ids under Custom model id… — that is a permission symptom, not a gateway failure.
• litellm must be installed. The shipped container image includes it. A custom install without the agent extra fails on the first real model call; the mock path never touches it.

Troubleshooting

Changes made on Platform defaults are recorded in the cross-tenant audit log as platform.model_tiers_default. The environment settings in this chapter are not audited — they leave no trace inside the product, so record them in your own change management.

Next steps

• Next: Upgrade and release verification
• All Install and operate the platform pages
• Documentation home