# Grafana

> Connect Grafana with a Viewer service-account token so the agent reads dashboards and queries the backends behind them, and point Grafana alerting at the workspace webhook.

_How-to · Updated 2026-10-23 · Describes release 0.1.164_

## Overview {#overview}

Grafana is an umbrella: one connection reaches the dashboards and, through Grafana's datasource proxy, the metrics, logs and traces backends behind them. The catalog therefore lists it under Alerting, Metrics, Logs, Tracing and Dashboards, and a connected Grafana counts as metrics, logs and traces coverage for the **Incident** agent; the **SecOps** and **FinOps** agents, the estate x-ray and the insights chat read it too.

The connection is the read side. Grafana alerting reaches the platform through a webhook contact point, see [Alerts in](#alerts-in).

## Setup {#setup}

### Before you start {#before-you-start}

- The `datasources.manage` permission; the built-in owner and operator roles carry it.
- A Grafana service-account token with the Viewer role ([Installation](#installation)).
- The Grafana URL reachable from the platform or from a [private-network agent](/help/private-access.md).

### Installation {#installation}

In Grafana open **Administration → Users and access → Service accounts** and add an account named `aiops-reader` with the **Viewer** role, then **Add service account token** and copy the token. Restrict the account's folders and datasource query permissions to the dashboards and backends the agent needs. A fix never runs through this connection, so there is no reason to raise it above Viewer.

### Configuration {#configuration}

1. Open **Settings · Data in → Data sources** and select **Connect datasource**.
2. Under **Type**, choose **Grafana**. Enter a **Connection name**; the connected list shows it, and per-connection agent tools are named after it.
3. Fill the fields; the TLS-skip field sits under **Advanced connection settings**:

| Field (portal label) | API key | Value |
|---|---|---|
| Grafana URL | `url` | `https://grafana.example.com` |
| API token (Viewer role) | `api_key` | The service-account token; stored write-only |
| Skip TLS verify for a self-signed cert (optional — type: true) | `insecure` | `true` only for a self-signed certificate |

4. Optionally, open **Optional · describe this environment** and fill **What should AIOps know about it?** with the dashboards and datasource names that matter.
5. Under **Use this connection for**, choose **Investigate only** and select **Connect and verify**.

The same connection through the API (`POST /api/aiops/{tenant_id}/datasources`, requires `datasources.manage`; the token goes in `secret_values`):

```json
{
  "type_key": "grafana",
  "name": "prod-grafana",
  "description": "Prod dashboards; the Checkout and Payments folders are the ones on call reads.",
  "config": {
    "url": "https://grafana.example.com"
  },
  "secret_values": {
    "api_key": "<service-account token>"
  },
  "role": "read",
  "enabled": true
}
```

### Validation {#validation}

**Connect and verify**, and **Test** on the row later, sends a GET to `/api/health` with `Authorization: Bearer <token>`. **You should see** a dialog titled `<connection name> is connected` with **Connected — verified from the control plane**, and later in the row's **Last test** `ok · HTTP 200 /api/health — {"commit": …`. Grafana answers that endpoint without authentication, so a green result proves the URL and TLS, not the token: a wrong token still passes **Test** and fails on the agent's first real request. Select **Audit perms** on the row to see what the token can read, or let the agent open one dashboard.

### If something fails {#if-something-fails}

The dialog shows **Needs a fix**, what to fix and the check's own detail; the row is saved either way. Select **Edit**, correct the field the detail names and save again, or select **Retry** once a cause outside the form (network, vendor-side permission) is fixed. Every verdict this connector produces, and what to do about it, is in [Troubleshooting](#troubleshooting) below.

## Data collected {#data-collected}

### Tools the agent gets {#tools}

- `<name>_request` — the HTTP tool against the base URL with `Authorization: Bearer <token>`: `method`, `path`, `query`, `body`, `content_type`. Dashboards come from `GET /api/search` and `GET /api/dashboards/uid/<uid>`, annotations from `GET /api/annotations`, the backends behind Grafana from `POST /api/ds/query` with the datasource uid in the body.

There is no native Grafana tool, and a Grafana connection does not register `prometheus_query`; a direct Prometheus connection does. Responses are capped at 20,000 characters per call.

### Read surface {#read-surface}

`GET` and `HEAD` run freely, and so do `POST` requests to `api/ds/query` and `api/tsdb/query`, the two query proxies. Every other `POST`, `PUT`, `PATCH` and `DELETE` — creating annotations or dashboards, silencing alerts, editing datasources — is refused in an investigation, never offered for approval: `Investigation connections are read-only. This operation was not executed; prepare an incident change using a write connection and obtain approval.` The full rule table is under **The read/write gate in detail** in [Connections and remediation access](/help/read-and-write-roles.md#read-role-vs-write-role).

### Alerts in {#alerts-in}

Grafana unified alerting posts its native, Alertmanager-shaped payload. Add a contact point of type **Webhook** with the URL `https://<your-host>/webhook/grafana/<workspace-slug>` and the token as `Authorization: Bearer <webhook-token>`; keep resolved notifications on, because `status` per alert is what closes the incident. Add a `service` label to each alert rule's **Labels** section — Grafana forwards custom labels verbatim, and `service` is the identity the platform groups on. The endpoint and the token are in the [webhook reference](/help/webhook-reference.md#url-forms-and-authentication). Field by field, with the response counters and error strings: [webhook reference](/help/webhook-reference.md#grafana).

## Troubleshooting {#troubleshooting}

| What you see | Cause | What to do |
|---|---|---|
| `failed · SSRF guard: '<host>' resolves to non-public address … — blocked. The agent can still use this endpoint — only the Test probe is blocked by the SSRF guard's secure default.` | A cluster-internal or private address. The agent's own tools are not SSRF-guarded. | Accept it, add the host to the workspace's `ssrf_allow_hosts`, or turn the workspace SSRF guard off under Guardrails. |
| **Test** is green, but every agent request answers `HTTP 401` | `/api/health` needs no token; the token is wrong, expired or revoked. | Rotate the token on **Edit** (a blank secret field keeps the stored value; only a non-empty one replaces it) and **Test** again. |
| Dashboards list, but `POST /api/ds/query` answers `HTTP 403` | The service account may see the dashboard but not query its datasource. | Grant the account datasource query permission in Grafana. |
| `failed · HTTP 404 /api/health` | The URL carries a path Grafana does not serve the API under. | Use the Grafana root URL, with the sub-path only if Grafana is served under one. |
| `failed · ConnectError: …` or `ConnectTimeout` | The platform cannot reach the URL. | Use an address the platform reaches, or install a [private-network agent](/help/private-access.md). |
| A `POST /api/annotations` or `POST /api/dashboards/db` answers `Investigation connections are read-only.` | The write gate. | Expected. |
| Grafana alerts arrive but open incidents with no service | The alert rule has no `service` label. | Add it under the rule's **Labels**, or map the rule's own label with `label_map`. |
| Grafana's webhook test answers `401 invalid webhook credentials` | Wrong token, wrong workspace slug or a disabled workspace; the endpoint never says which. | Mint or copy the token under **Alert intake & tokens** and paste the exact URL from there. |

The generic verdict strings and the SSRF guard are explained in [Test a connection and read the failure](/help/manage-connections.md#troubleshooting). Every connection status word is in the [status and error reference](/help/status-and-error-reference.md#connection-statuses).

## Next steps {#next-steps}

- Select **Discover this connection** in the success dialog to find the services and dependencies behind it: [Launch a scan](/help/estate-harvest-and-xray.md#launch-a-scan).
- Send Grafana alerts in through the workspace webhook: [Alerts in](#alerts-in).
- See what else each agent needs in [Choose your connection](/help/choose-your-connection.md#agent-needs).