# Set up the Test Lab on your own cluster

> Provision an isolated test environment using your own Kubernetes cluster.

_How-to · Updated 2026-10-17 · Describes release 0.1.164_

## What you are about to build

You give the platform **one kubeconfig**. It installs six demo estates, a
full observability stack, and wires that stack's alertmanager to post alerts
back into your Test Lab workspace. Nothing here asks you to write a manifest.

When it finishes, the lab looks like this:

| Namespace | What lands there | Pods |
| --- | --- | --- |
| `hotel` | DeathStarBench hotelReservation | 21 |
| `boutique` | Google Online Boutique | 12 |
| `socialnetwork` | DeathStarBench socialNetwork | 27 |
| `pgdemo` | a small Postgres and the app that pools against it | 3 |
| `logsearch` | single-node OpenSearch and a log writer | 2 |
| `edge` | an SSH server that answers like a network device | 1 |
| `observability` | Prometheus, Alertmanager, Loki, Tempo, Grafana, promtail, kube-state-metrics, blackbox | 10 |

The estates are real applications with real dependency chains, not stubs —
that is the point. A case that kills a cache has to produce the same confusing
symptoms your production incidents do, or the accuracy number it earns means
nothing.

## Before you start

Creating the lab workspace needs `members.manage`. Connecting, validating,
provisioning, changing guards and teardown need `datasources.manage`.
Launching or cancelling live runs and replays needs `benchmark.manage`;
reading results and certifying needs `benchmark.view`.

**A cluster you are willing to break.** Faults are injected on purpose: pods
are killed, deployments are scaled to zero, images are corrupted, network paths
are cut. Use a cluster that hosts nothing you care about.

- **Kubernetes 1.27 or newer.**
- **Free capacity: about 6 vCPU and 8 GiB**, on top of anything else running.
  That is the six estates plus observability at rest. Add headroom if you
  plan to run cases in parallel — each extra shard is another copy of `hotel`.
- **A default StorageClass.** Each copy of `hotel` (the base and every shard)
  claims one 1 GiB volume for `mongodb-rate`, so the volume-full cases have a
  real claim to fill. The claims live in the estate namespaces and go with them
  on teardown. Every other estate is ephemeral.
- **Outbound image pulls** to `us-central1-docker.pkg.dev`, `docker.io` and
  `gcr.io`. All images are public; none are ours.
- **No ingress controller, no load balancer, no public DNS.** The lab talks to
  itself over cluster DNS, and the platform reaches it through your kubeconfig.

## Step 1 — a kubeconfig the platform can build with

Provisioning creates namespaces, deployments, services, cluster roles and
custom resources, so this credential has to be **cluster-admin**. The platform
checks: it runs `kubectl auth can-i '*' '*'` and refuses to start if the answer
is anything but yes. A narrower role fails before the first manifest is
applied, rather than half way through.

```bash
kubectl create namespace lab-access
kubectl create serviceaccount lab-provisioner -n lab-access
kubectl create clusterrolebinding lab-provisioner \
  --clusterrole=cluster-admin \
  --serviceaccount=lab-access:lab-provisioner

# a non-expiring token: a projected one would expire mid-run
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: lab-provisioner
  namespace: lab-access
  annotations:
    kubernetes.io/service-account.name: lab-provisioner
type: kubernetes.io/service-account-token
EOF
```

Then build a kubeconfig from it:

```bash
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl -n lab-access get secret lab-provisioner -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n lab-access get secret lab-provisioner -o jsonpath='{.data.token}' | base64 -d)

cat > lab-kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
clusters: [{name: lab, cluster: {server: ${SERVER}, certificate-authority-data: ${CA}}}]
users: [{name: lab, user: {token: ${TOKEN}}}]
contexts: [{name: lab, context: {cluster: lab, user: lab}}]
current-context: lab
EOF
```

**If the platform runs inside the same cluster**, use
`https://kubernetes.default.svc` as the server instead of the public API
endpoint — the call never leaves the cluster.

## Step 2 — register it as a write-role datasource

Open **Agent quality → Test Lab**, and on the **Lab cluster** card choose **Connect lab cluster**.
The form opens on **Kubernetes (kubectl + kubeconfig)** with the role fixed to
**Lab provisioning and fault tests · write credential**: paste the file, then **Save lab connection**.
No automatic read probe runs; the saved credential's access is not yet verified.
For this Test Lab setup, continue to **Provision** below: the provisioning
operation and its log show whether the required write access works.

The role matters more than it looks. A `write` datasource is a
remediation-class credential: it is excluded from every agent registry, so no
investigation, x-ray scan or chat turn can see it or use it. Only the Test Lab's
own provisioning path executes with it. The Lab refuses a `read` datasource for
exactly this reason — a cluster-admin token an agent could reach for is a
different product.

## Step 3 — provision

**Test Lab → Provision**, choose the datasource, and confirm. A few minutes.
This is the only click; everything in the next section happens inside it.

Before applying anything the platform lists every namespace on the cluster and
flags the ones a lab does not normally have. That list is advisory — it exists
so you notice *before* the estates land if you pointed at the wrong cluster.

Three settings decide whether alerts actually come back. Get these right and
everything else is automatic:

- **Reachable platform URL.** The lab's alertmanager has to POST to this
  platform. If the lab sits behind a firewall, or this platform only accepts
  traffic from a CDN, the default public URL will not work — supply the address
  the *lab* can reach, which for a same-cluster lab is the internal service
  name.
- **The ingest token.** If your workspace already has a webhook token, hand it
  over here. Provisioning will not rotate a token in use: rotating it would
  401 every real alert source you already have.
- **Parallel shards.** Leave at 1 unless you are running the catalogue for
  time. Each shard is a full copy of `hotel`. A re-provision keeps the count
  you set last, and deletes any shard above it that carries the lab's estate
  label; one it cannot prove is the lab's stays, and the log names it.

## What Provision does for you

Nothing below is a step you take. It is what the one click does, listed so you
know what lands on your cluster:

- **The six estates and the observability stack**, as above.
- **Alertmanager rewritten** to post at this platform, then restarted so it
  reads the new config.
- **Chaos Mesh and LitmusChaos**, installed from the versions this platform
  ships, each only when the cluster does not run it yet. One that is already
  running is left alone: a re-provision must not restart the injector under a
  campaign in flight. LitmusChaos counts as running when its operator does —
  its CRDs alone are not enough.
- **An event guard on the estates.** An admission policy, matched to the
  estate namespaces only, refuses the chaos engines' own events there, so the
  agent under test cannot read which fault was injected; every other event
  passes. It needs Kubernetes 1.30 or later. Without it the provision still
  succeeds, and runs remove those events while each case is in flight — the
  log says which applies.
- **The lab Git server**, for the lab's GitOps cases; its data is ephemeral.
- **The lab cluster connection, scoped to the estates.** Typed restarts the
  change agent proposes may target any estate namespace, every `hotel` shard
  included. A **Test** on that connection confirms only the namespace its
  kubeconfig context names, if any, until the next provision.
- **Your lab datasources, registered.** The read-only Kubernetes identity, plus
  Prometheus, Loki, Tempo and Alertmanager. Each URL is probed first and skipped
  if the platform cannot reach it, so you never end up with a datasource that
  looks connected and is not.

Two things to know about that last one. These are cluster-internal addresses,
so they only work when this platform runs in the same cluster as the lab; if it
does not, the provision log names each one it skipped and you add reachable
addresses yourself. And the hostnames it did register are added to your
workspace's allowed-hosts list — without that, the datasource works for the
agent but its **Test** button reports the SSRF guard, because the guard's job is
to stop anyone pointing a datasource at an internal address that was never
meant to be reached.

If a chaos install fails, the provision still succeeds. The estate is up and the
cases that scale, patch and delete still run; only fault injection is missing,
and the log says so. Re-run provision to retry it.

## Verify

- **Test Lab → Provision log** ends with `provision complete — estate is
  feeding alerts at …`.
- All four namespaces report every pod running.
- Within a few minutes, **Incidents** is quiet. If a "storefront is not
  answering" incident opens and stays open, an estate did not come up — the
  probe is telling you which one.
- **Settings · Data in → Data sources** lists the lab's read connections; review their read-test results. The separate write credential is checked by the provisioning operation, not by a green read test.

Then run two or three cases before the full catalogue. A run that scores zero
usually means the alerts never arrived, not that the agent was wrong.

## Tearing it down

**Test Lab → Teardown** removes what provisioning created, and only that:

- the estate namespaces — every `hotel` shard the lab made included —
  `observability` and the lab's Git server (a shard-named namespace without
  the lab's estate label stays, and the log names it)
- the read-only roles it created outside any namespace, for the agent and the
  metrics scraper, and the estates' event guard
- LitmusChaos, when provisioning installed it (its CRDs stay; the log shows
  the command that removes them)
- the datasources it registered for you, and their entries in your
  allowed-hosts list

It does **not** touch your lab kubeconfig, past runs, or their graded results.
A namespace still terminating three minutes after its delete ends the job
**done** with a WARNING naming it: the delete was accepted, so run Teardown
again to confirm it is gone. When a step fails — a delete the cluster refused,
a namespace still terminating on that second run or whose deletion the
cluster reports blocked, a route of yours still using a lab connection — the
job ends **failed** and names what is left. Fix the cause and run Teardown
again: it picks up where the last one stopped.

**Chaos Mesh is deliberately left installed.** Provisioning only installs it
when the cluster has none, so it may well have been there before this lab —
and it is cluster-scoped: uninstalling deletes the chaos CRDs, and with them any
experiment anyone else on that cluster is running. The teardown log shows the
command instead, with a copy button, so removing it is a decision you make:

```bash
helm uninstall chaos-mesh -n chaos-mesh &&   kubectl delete ns chaos-mesh &&   kubectl get crd -o name | grep chaos-mesh.org | xargs -r kubectl delete
```

The CRD line matters — `helm uninstall` leaves custom resource definitions
behind by design. If Argo CD installed Chaos Mesh there is no helm release, and
self-heal puts back whatever you delete: remove its Application from Git
instead, after adding the `resources-finalizer.argocd.argoproj.io` finalizer
to it, or the prune leaves the workloads and CRDs running.

Re-provisioning is idempotent, so tearing down between campaigns to stop paying
for the nodes is a reasonable habit.


## Next steps

- Next: [Benchmarks and scores](/help/benchmarks-and-scores.md)
- [All Work with the agents pages](/help/work-with-the-agents/llms.txt)
- [Documentation home](/help/)