# Prometheus

> Connect Prometheus, Mimir, Cortex, Thanos or VictoriaMetrics for PromQL reads, what the agent's prometheus_query tool returns, and how its reads are gated.

_How-to · Updated 2026-10-23 · Describes release 0.1.164_

## Overview {#overview}

A Prometheus connection gives the agents PromQL. The **Incident** agent reads it during diagnosis (metrics is one of the five categories that agent needs) and the **FinOps** agent reads utilisation baselines from it; the estate x-ray and the insights chat use the same connection. One connection also feeds two more things: the hourly service-map sync reads the Tempo service-graph metric stored in Prometheus, and a Lakehouse collection can pull metrics from it on a schedule.

The connector speaks the Prometheus HTTP API, so the same type covers Mimir and Cortex (set a path prefix and tenant ids), Thanos and VictoriaMetrics (leave both blank). Prometheus alerts do not arrive through this connection: Alertmanager posts them to the workspace webhook, see [Alerts in](#alerts-in).

## Setup {#setup}

### Before you start {#before-you-start}

- The `datasources.manage` permission; the built-in owner and operator roles carry it.
- No credential in the form; a backend behind authentication takes one through the REST API (see [Installation](#installation)).
- The query API exposed to the platform or to a [private-network agent](/help/private-access.md); [Installation](#installation) lists the paths.

### Installation {#installation}

Expose the query API to the platform or to a [private-network agent](/help/private-access.md): `api/v1/query`, `api/v1/query_range`, `api/v1/series`, `api/v1/labels` and `api/v1/status/buildinfo`, plus `/-/healthy` for the connection check. Keep the admin API (`api/v1/admin/tsdb/*`) and remote write out of the exposed surface; the platform refuses them anyway, but a gateway that does not serve them is the real boundary.

The form has no credential field. A Prometheus behind authentication takes its credential through the REST API: `secret_values.api_key` is sent as `Authorization: Bearer <token>`, or `secret_values.username` and `secret_values.password` as HTTP Basic. Both are stored write-only and merged into the connection at use.

### Configuration {#configuration}

1. Open **Settings · Data in → Data sources** and select **Connect datasource**.
2. Under **Type**, choose **Prometheus**. Enter a **Connection name**; the connected list shows it, and per-connection agent tools are named after it.
3. Fill the fields; the TLS-skip field sits under **Advanced connection settings**:

| Field (portal label) | API key | Value |
|---|---|---|
| Prometheus base URL | `url` | The query service, for example `https://prometheus.example.com` or `http://prometheus.monitoring.svc:9090` inside the cluster |
| API path prefix (Mimir/Cortex: /prometheus; Prometheus/Thanos/VictoriaMetrics: leave blank) | `path_prefix` | `/prometheus` for Mimir and Cortex; blank otherwise |
| X-Scope-OrgID tenant id(s), comma-separated (Mimir/Cortex multi-tenant; leave blank otherwise) | `org_ids` | `team-a, team-b` for a multi-tenant backend |
| Skip TLS verify for a self-signed cert (optional — type: true) | `insecure` | `true` only for a self-signed certificate |

4. Optionally, open **Optional · describe this environment** and fill **What should AIOps know about it?** with what a new on-call engineer would need.
5. Under **Use this connection for**, choose **Investigate only** and select **Connect and verify**.

The same connection through the API (`POST /api/aiops/{tenant_id}/datasources`, requires `datasources.manage`):

```json
{
  "type_key": "prometheus",
  "name": "prod-prometheus",
  "description": "Prod cluster metrics; checkout and payments are the critical services.",
  "config": {
    "url": "https://prometheus.example.com",
    "path_prefix": "",
    "org_ids": ""
  },
  "role": "read",
  "enabled": true
}
```

For Mimir, send `"path_prefix": "/prometheus"` and `"org_ids": "team-a, team-b"`; for a backend behind a bearer token add `"secret_values": {"api_key": "<token>"}`.

### Validation {#validation}

**Connect and verify**, and **Test** on the row later, sends a GET to `/-/healthy` and then to `/api/v1/status/buildinfo`. With a path prefix it sends only `<prefix>/api/v1/status/buildinfo`, because `/-/healthy` is not served under the prefix; with tenant ids it sends `X-Scope-OrgID` with the first tenant. **You should see** a dialog titled `<connection name> is connected` with **Connected — verified from the control plane**, and later in the row's **Last test** `ok · HTTP 200 /-/healthy — Prometheus Server is Healthy.`: the status, the path answered and the first 120 characters of the body. The check proves reachability and, where the backend enforces it, authentication; it does not prove that the series you expect exist. Run `up` through the agent, or in your own Prometheus, and confirm the intended services appear.

### If something fails {#if-something-fails}

The dialog shows **Needs a fix**, what to fix and the check's own detail; the row is saved either way. Select **Edit**, correct the field the detail names and save again, or select **Retry** once a cause outside the form (network, vendor-side permission) is fixed. Every verdict this connector produces, and what to do about it, is in [Troubleshooting](#troubleshooting) below.

## Data collected {#data-collected}

### Tools the agent gets {#tools}

- `prometheus_query` — bound to the first Prometheus connection in the workspace. Parameters: `query` (PromQL), and `start`, `end`, `step` for a range query; an instant query runs when neither `start` nor `end` is given. With more than one tenant id, `org_id` chooses the tenant (the first is the default). The response is compacted into `metric{labels} = value` lines: at most 50 series, the last 20 points of each range series (`+N earlier points` says what was cut), `success: empty result set (no matching series)` when nothing matched, and `HTTP <code>` with the backend's body on an error.
- `<name>_prometheus_query` — every further Prometheus connection gets its own native, named after the connection, so a fragmented estate is fully reachable.
- `<name>_request` — the raw HTTP tool against the base URL: `method`, `path`, `query`, `body`, `content_type`, and `org_id` when tenant ids are set. Use it for the label and series endpoints.

Responses are capped at 20,000 characters per call.

### Read surface {#read-surface}

`GET` and `HEAD` run freely, and so do `POST` requests to `api/v1/query`, `api/v1/query_range`, `api/v1/series`, `api/v1/labels` and `api/v1/query_exemplars`, matched on whole path segments so `/prometheus/api/v1/query` qualifies too. Everything else is refused in an investigation, never offered for approval: `POST /api/v1/write`, and the admin endpoints `api/v1/admin/tsdb/delete_series`, `clean_tombstones` and `snapshot` even over `GET`, because VictoriaMetrics answers a delete on `GET`. The refusal reads `Investigation connections are read-only. This operation was not executed; prepare an incident change using a write connection and obtain approval.` The full rule table is under **The read/write gate in detail** in [Connections and remediation access](/help/read-and-write-roles.md#read-role-vs-write-role).

### Alerts in {#alerts-in}

Alerting rules reach the platform through Alertmanager's webhook receiver, `POST /webhook/alertmanager/<workspace-slug>` with the token in `Authorization: Bearer` and `send_resolved: true`, not through this connection. Every alerting rule must carry a `service` label: set it on the rule, or attach it in the scrape config so it rides every series. [Alerts from your observability platform](/help/external-observability.md) covers the setup; the [webhook reference](/help/webhook-reference.md#alertmanager) has the fields the adapter reads.

### Service map {#service-map}

When Tempo's service-graph processor remote-writes `traces_service_graph_request_total{client, server}` into this Prometheus, the hourly trace sync reads `increase(traces_service_graph_request_total[3600s])` summed by `client`, `server`, `connection_type` and the scope labels `namespace`, `cluster` and `k8s_namespace_name`, once per declared tenant id, and records caller → callee edges with origin `traced`. Series whose `connection_type` is `messaging_system` or `virtual_node` are dropped. Traced edges show on the map and widen the blast radius; they fold incidents only where the workspace opts traced edges in (`aiops_topology_fold_origins`).

## Troubleshooting {#troubleshooting}

| What you see | Cause | What to do |
|---|---|---|
| `failed · SSRF guard: '<host>' resolves to non-public address … — blocked. The agent can still use this endpoint — only the Test probe is blocked by the SSRF guard's secure default.` | A cluster-internal or private address. The agent's own tools are not SSRF-guarded. | Accept it, add the host to the workspace's `ssrf_allow_hosts`, or turn the workspace SSRF guard off under Guardrails. |
| `failed · HTTP 401 /-/healthy` or `HTTP 403` | The backend wants a credential; the form has none. | Add `secret_values.api_key`, or `username` and `password`, through the API. |
| `failed · HTTP 404 /prometheus/api/v1/status/buildinfo` | The path prefix does not match how the backend serves its API. | Leave the prefix blank for Prometheus, Thanos and VictoriaMetrics; `/prometheus` is the Mimir and Cortex default. |
| `failed · ConnectError: …` or `ConnectTimeout` | The platform cannot reach the URL. | Use an address the platform reaches, or install a [private-network agent](/help/private-access.md) where the backend lives. |
| A certificate error in the detail | A self-signed certificate. | Set **Skip TLS verify** to `true`, or install a trusted certificate. |
| `prometheus_query` answers `HTTP 400` with `"errorType":"bad_data"` | The PromQL did not parse. | Read the backend's message in the output; the agent usually retries with a corrected query. |
| `success: empty result set (no matching series)` on a query that works in Grafana | The wrong tenant, or the series have other labels. | Check `org_ids` and the default tenant; compare label names with `<name>_request` on `/api/v1/labels`. |
| A call to `api/v1/admin/tsdb/delete_series` or `api/v1/write` answers `Investigation connections are read-only.` | The write gate. | Expected. A fix never runs through an investigation tool. |
| A VictoriaMetrics connection has no `prometheus_query` | The **VictoriaMetrics** type registers only the HTTP tool. | Connect the same URL as a **Prometheus** connection to get the native PromQL tool. |
| The agent never uses the connection | Disabled, write role, a pillar allowlist that excludes the agent, or a type platform staff disabled. | Check **State**, **Use this connection for** and **Pillar allowlist** on **Edit**. |

The generic verdict strings and the SSRF guard are explained in [Test a connection and read the failure](/help/manage-connections.md#troubleshooting). Every connection status word is in the [status and error reference](/help/status-and-error-reference.md#connection-statuses).

## Next steps {#next-steps}

- Select **Discover this connection** in the success dialog to find the services and dependencies behind it: [Launch a scan](/help/estate-harvest-and-xray.md#launch-a-scan).
- Send Prometheus alerts in through Alertmanager's webhook: [Alerts in](#alerts-in).
- See what else each agent needs in [Choose your connection](/help/choose-your-connection.md#agent-needs).