How it works

A runbook is a Markdown file in docs/runbooks/ that proves one flow: its preconditions, its ordered steps, and the exact result each step must show. The agent writes runbooks from your code, runs them and records each result. Each runbook has one surface. A ui runbook runs in the browser. An api runbook runs through a test script in your test library. The agent reads no database and writes no SQL. Only the browser or the script decides a result.

The flow

You set up a project once. After that, each flow gets its runbook through create and run, and the runbook changes again only when the behavior it records changes. The checks against the code and the page with the results run whenever you want.

/runbooks:orchestrator takes many runbooks through this flow at once. You give it a goal, such as every runbook of one capability, and it launches agents that map, create, run and script them in parallel, asks you what only you can decide, and commits each runbook when it reaches the goal. See Orchestrator.

The life of a runbook

A runbook's status says how far it got. Coverage counts a route or endpoint as covered only through an active runbook.

A placeholder has a title and a goal but no steps. A draft has steps but never passed a full run. An active runbook that fails later stays active, and last_run shows the failure. A deprecated runbook stays for history, and --affected never selects it.

A step the user waives counts as a pass, provided at least one other step passed. A run limited with --steps gets a report but leaves status and last_run unchanged.

What a runbook holds

The frontmatter says what the runbook covers. The body holds the steps. Each ui step names the route it starts on, the controls by their visible labels, and the exact text the page must show. A --- line goes before each section and before each step after the first.

---
id: billing.issue-invoice
title: Issue a client invoice
capability: billing
surface: ui
roles: [admin]
viewports: [desktop, mobile]
covers:
  entries: ["/billing/invoices", "POST /admin/internal/invoices"]
status: draft
last_run: null
---

# Issue a client invoice

---

## Steps

### S01. Admin opens the invoice list

Parents: none
Page: `/billing/invoices`

Actions:

1. Click "Invoices" in the sidebar.

Expected: the heading "Invoices" and a table with the columns "Client", "Amount", "Status".

An api step names an Endpoint: and the role it calls As:. From runbooks 1.9.2, Request: is an http code block with the method, the path, the query, the headers that matter and the JSON body. Expected: gives the status on its line, then a json code block with the body fields the step checks.

### S01. Admin lists the airports

Parents: none
Endpoint: `GET /quotes/airports`
As: admin
Request:

```http
GET /quotes/airports?page=1&perPage=50&sortBy=name&sortOrder=asc
```

Expected: 200.

```json
{
  "meta": { "total": 2, "page": 1, "perPage": 50, "totalPages": 1 },
  "data": [
    { "id": "<uuid>", "iataCode": "MIA", "name": "Miami International" },
    { "id": "<uuid>", "iataCode": "GRU", "name": "São Paulo/Guarulhos" }
  ]
}
```

• The script checks only the keys the block shows.
• An array lists its items in the order the response returns them, and the script checks that order and the length. A sentence after the block says when only some items matter or the order does not.
• A value in angle brackets is checked by its type or its source, not as text: "<uuid>", "<ISO 8601>", "<id of the draft invoice>". It stays a JSON string, so the block is valid JSON.
• A response with no body, such as a 204, has no block. Credentials stay out of the request: As: or the role lines name who calls, and the project's test helper signs the call.

An api runbook covers at least the main path, invalid input (422), a role that must not get in (403), no authentication (401), and a read back of anything it saves. With role tracking on, the 403 and 401 cases become role lines on the main step (see Roles). /runbooks:audit flags steps written before 1.9.2 as old-format, and --fix rewrites them in blocks with the same checks.

What happens in a run

/runbooks:run splits the selection by surface. A ui runbook runs once per viewport it lists, such as desktop then mobile, and compares each page with its step's Expected. An api runbook runs its test script, which /runbooks:to-test writes or adopts when scripts is empty. Each test name starts with its step id (S01 admin creates a draft), so each result lands on its step. Both kinds end in the same report and the same status rules.

The agent runs every step before it fixes anything, so one run lists every failure. A difference from the design goes in the report as major or minor and never changes a step's state: only Expected decides it.

Steps that depend on steps

Each step lists its Parents:, the steps whose failure makes its own result meaningless. A step runs only when every parent passed. When a parent fails, the agent skips the whole branch below it and marks those steps blocked-by-parent, so one bug shows up as one failure. The runbooks page draws this tree for each runbook.

The runbooks page

/runbooks:ui starts a server on 127.0.0.1, port 4020 by default, and gives you its URL. The page reads the runbooks, the reports in _runs/ and the config each time it loads, so a new run or an edit shows when you reload. It only reads files. It changes none and starts no run.

❯ /runbooks:ui
❯ /runbooks:ui --port 4800 # when 4020 is taken

The Runbooks section has one sidebar item per capability, each with a red count of its problems: stale runbooks and entry points no runbook covers. A capability shows its runbooks with the date, time and result of their last full run, then the uncovered routes of the slice with the same name. A slice with no runbook yet shows up too, so every uncovered entry point is on the page.

A runbook is stale when its last full run is more than staleDays days old, or when it never ran. staleDays is 30 unless you change it with /runbooks:settings stale. Only active and draft runbooks go stale. A run with --steps does not reset the age, because only a full run writes last_run.

When a capability has a ui runbook, its table gets an Upstream column with each runbook's mode, live or mocked. The ? next to the column name explains both.

A runbook's page shows its frontmatter, its calls included, and its steps, each with its state from the last full run, one per viewport. The History tab lists every report of that runbook in _runs/, newest first. A report opens on its own page, with a breadcrumb back to the history. Git ignores the reports, so the history holds the runs of your machine only. When last_run comes from a teammate's run, the page says "report not on this machine" and the steps show no state.

From runbooks 1.9.4, a runbook's page draws its steps as a dependency tree, built from their Parents: lines, right before the steps. A step with two parents shows under each one, marked "also after" the other. A parent that is not in the runbook counts as none, and a cycle in Parents: shows as "cycle" instead of looping. When no step depends on another, the page says the steps can run in any order.

The Coverage section shows the numbers of /runbooks:coverage, overall and per slice. A slice lists each entry point with its state, the runbooks that cite it and the scripts that test it. The scopes of /runbooks:coverage have addresses too: #/coverage?path=<glob>, #/coverage?feature=<y> and #/coverage?capability=<x>, and /runbooks:coverage gives the address of the scope it reported. This section runs the requirement sources of the config on every load, command sources included. The ranked list of runbooks to write next stays in /runbooks:coverage.

On tablets and phones, the sidebar closes into a drawer that a menu button opens.

Live and mocked APIs

The screens of a ui runbook call an API, the upstream API. The runbook's upstream mode says how it reaches that API. A live runbook calls the real API, which holds test data. A mocked runbook gets fixed responses, its fixtures, from the upstream server, a file that setup copies into your project.

upstream.mode in the config sets the project default, and a runbook overrides it with upstream: live or upstream: mocked in its frontmatter. A config without upstream means live. Every run mode works with both, and the design comparison runs in both.

A mocked runbook never proves what the API saves. A step that would check persistence checks the request instead (method, URL and body), and persistence goes under "Not covered". The agent reads that request from the server's request log.

One fixture is one JSON file: the request it answers and the response it sends. source says where it came from, and the run report counts the fixtures that were not recorded.

{
  "request": { "method": "GET", "path": "/v1/invoices", "query": { "page": "1" } },
  "response": {
    "status": 200,
    "headers": { "content-type": "application/json" },
    "body": { "items": [] }
  },
  "source": "recorded 2026-10-20 from https://api.example.com"
}

create writes the fixtures with the runbook, and update records them again when the steps change. To record them yourself, run the flow once against the live API:

❯ /runbooks:run billing.issue-invoice --record --mode headless

The recorder drops authorization, the cookies and every header in upstream.redactHeaders. From request bodies and GraphQL variables, it drops every key whose name contains password, token, secret or a name in upstream.redactBodyKeys, at any depth. Show the diff of the fixtures before you commit them. A request that matches no fixture gets a 501 that names the method, the path, the GraphQL operation and the payload it carried, its step becomes blocked, and run offers /runbooks:update to record it.

To move a runbook to the other mode, run /runbooks:update. Going to mocked records the fixtures and turns persistence steps into request checks. Going to live deletes the fixtures and restores those steps.

GraphQL and request bodies

A GraphQL app sends every operation as POST /graphql, so the method and the path cannot tell one query from another. From runbooks 1.11.0, a fixture names the operation it answers in request.operationName, and a runbook holds one fixture per operation its page sends. A precondition such as "the client belongs to no group" is then the fixture of the operation that reads the groups.

{
  "request": {
    "method": "POST",
    "path": "/graphql",
    "operationName": "SearchClients",
    "variables": { "type": "clients" }
  },
  "response": { "status": 200, "body": { "data": { "searchClients": [] } } },
  "source": "hand-written: clients tab"
}

From 1.12.0, a fixture can also match on the payload. request.variables tells apart two calls of one operation, such as the tabs of a page that all send SearchClients. request.body does the same for a REST request with a JSON body. The match is partial: every key the fixture names must be in the request with the same value, at any depth, and the request may carry more. An array or a scalar matches only an equal value. When several fixtures match, the one in the current step's folder wins, then the one that names the operation, then the one that names more fields.

• A body that is a JSON array of operations is a batch. Each item gets the fixture of its own operation, and the server answers with the results in batch order. One failing fixture, such as a 500 for one operation, fails the whole batch, as a GraphQL server does.
• The recorder writes the operation into the fixture and its file name, as post--graphql-GroupsByPartyId.json, and saves the variables or the body after redaction, with a hash of them in the file name. Two calls with different variables write two files.
• A value that changes on every run, such as a timestamp or an id the client generates, makes a recorded fixture stop matching. Remove those keys from the recorded file by hand.

The copy in the project

From runbooks 1.14.0, the agent, your team and CI all start the same server: the copy that setup writes into your project. Setup makes the copy whenever the config has an upstream block, live-only projects included, and writes its path to upstream.serverFile:

• <tests.ui.dir>/runbooks-upstream.mjs, next to your UI tests;
• the same path inside the UI tests repository, when the UI tests live in one;
• docs/runbooks/runbooks-upstream.mjs, when the project has no UI tests.

Setup also adds a runbooks:upstream script to package.json. Started without --fixtures, the copy reads docs/runbooks/runbooks.config.json and takes the port, the fixtures folder, the live API and the redact lists from it, so anyone on the team starts it with one command:

❯ npm run runbooks:upstream

run, to-test and the orchestrator check the copy before the first step. When it is missing, comes from another plugin version, or sits in a UI tests repository that is not cloned, they stop and offer /runbooks:setup, which refreshes it. They never copy it themselves. The copy carries the plugin version, so every upgrade asks for a setup run, and /runbooks:upgrade offers one. CI jobs written before 1.14.0 pass --fixtures, and the copy then reads no config and behaves as before.

The server holds one runbook and one step for every client, so a suite that uses it runs with one worker. The upstream reference has the config keys, the matching rules and the control routes.

An API in another repository

When the upstream API lives in another repository, each repository has its own runbooks install, and neither reads the other's files. CI checks out one repository at a time, so a config that read a folder next to the project would report one set of numbers on your machine and another in CI. The API's install covers its endpoints through api runbooks. In this repository, a ui runbook lists the operations its screens reach in calls, as METHOD /path with {param} segments. Calls never count as coverage and never make an orphan.

covers:
  entries: ["/billing/invoices"]
calls: [GET /invoices, POST /invoices, GET /invoices/{id}]

The contract between the two repositories is the OpenAPI 3 JSON document the API publishes. This repository keeps a copy at upstream.schema, docs/runbooks/_contract/openapi.json by default. The plugin fetches nothing: a job in your CI brings each new version in and opens a pull request with it.

runbooks.mjs contract checks every call against the operations of the schema, and every fixture of a mocked runbook against its calls and the schema. Parameter names do not matter, so {id} matches {invoiceId}. Findings exit 0, and with --strict they exit 1, so a CI job can fail on them. In CI the plugin is not installed, so the job checks out the plugin's repository at the version the project uses and runs the script from the project root:

node <plugin checkout>/runbooks/scripts/runbooks.mjs contract --strict

When upstream.schema changes, /runbooks:list --affected compares it with the version at the base, operation by operation, and lists each runbook whose calls name a changed, added or removed operation, with a reason such as upstream schema: GET /invoices/{id} changed. Descriptions, examples, tags and x- keys change nothing. /runbooks:update then records the fixtures again. Without upstream.schema, nothing in this repository notices an API change before a live run.

A project set up before runbooks 1.4.0 may read the API through an entries source whose root is outside the project, such as "../api". list and coverage warn about it. /runbooks:settings sources, or /runbooks:setup run again, moves those ids from covers.entries to calls. It shows a dry run first, with the entry coverage before and after, and writes only after you approve. The endpoints of the other repository then leave this project's coverage. From runbooks 2.0, an entries source, a slices glob or a tests.api.root outside the project is a config error. An API that is a folder of this repository, as in a monorepo, stays an entries source.

UI tests in another repository

Some teams keep their end-to-end suite in a repository of its own: the app in project-001 and its Playwright tests in project-001-e2e, cloned next to it. From runbooks 1.9.0, setup asks where the UI tests live and writes the path of that repository, relative to the project, to tests.ui.root. Only the scripts move. The runbooks, the fixtures and the coverage stay in the app's repository, so CI and your machine report the same numbers whether the UI tests repository is cloned or not.

"tests": {
  "ui": {
    "root": "../project-001-e2e",
    "dir": "tests",
    "framework": "playwright"
  }
}

• tests.ui.dir is relative to tests.ui.root. A runbook's scripts stay relative to the project, such as ../project-001-e2e/tests/billing/pay.runbook.spec.ts, and run turns them into paths relative to the root before it calls the test command.
• Setup reads only the folder you name: its manifest, its framework config and two or three of its tests, so to-test follows their conventions. A framework installed then is installed in that folder.
• to-test and /runbooks:settings tests commit the files they write there in a commit of their own, in that repository, by the same policy as here. run in mode test and to-test stop when the folder does not exist and say where to clone it.
• Every developer clones the UI tests repository at the same path relative to the app. Its CI checks out the app too, at that path, to start the app and read the fixtures. The plugin writes no CI file.

The exception covers tests.ui.root only. An API in another repository still has its own install, and its api runbooks run there (see An API in another repository).

Roles and permissions

With role tracking on, a runbook runs its steps once per role, and each step says what each role should see: the result for an editor, a denial for a visitor with no session. The project keeps its roles and their permissions in roles.json, and changing a permission marks the runbooks it reaches for review. Roles explains the problem it solves and how it works.

Which model runs each command

From runbooks 1.13.0, the models table in the config names the model each command runs on, per command and per parameter. A row set to inline runs the command in the main session. Any other row, haiku, sonnet, opus or fable, makes the main session launch a sub-agent on that model. Mechanical work, such as reading a JUnit report in run --mode test, then runs on a smaller model, and the commands that write Expected text run on opus.

• A sub-agent cannot ask you anything. It reports its question, the main session asks you, and the answer goes back to the same sub-agent. When it ends, the main session prints its summary and commits once for the whole command.
• In run --mode test, a haiku sub-agent that finds a script missing, out of sync or broken does not fix it. It hands the runbook to to-test on its own row, and a new sub-agent runs it again. A second hand-off for the same runbook ends it as blocked.
• --model inline|haiku|sonnet|opus|fable on any command forces one model for that invocation and turns off the hand-off. ui accepts it and always stays in the main session, because its server must outlive the command.
• Setup writes the default table into the config. /runbooks:settings models changes a row. When a newer version changes a default, running setup again adds the new rows and asks which changed rows take the new default.
• When a model cannot launch, the command shows the error and the row it came from, offers --model and /runbooks:settings models, and stops. It never retries on another model.

The config reference has the matching rules, and the agents reference has the brief and the report each sub-agent follows.

Step by step

❯ /runbooks:setup

❯ /runbooks:map
❯ /runbooks:coverage # optional: a baseline before any runbook exists

❯ /runbooks:create --from placeholder billing.issue-invoice
❯ /runbooks:create --from ticket CHE-581 # or from a ticket, spec, diff or description

❯ /runbooks:run billing.issue-invoice --mode headless
❯ /runbooks:run --capability billing # every runbook of one capability

❯ /runbooks:check billing.issue-invoice # the latest result, without running again
❯ /runbooks:ui # every runbook, its runs and its reports in the browser

❯ /runbooks:update billing.issue-invoice --from diff main
❯ /runbooks:list --affected # the runbooks the changes since main touch
❯ /runbooks:run --affected --mode headless

❯ /runbooks:audit --all
❯ /runbooks:coverage --all-slices

❯ /runbooks:to-test billing.issue-invoice

❯ /runbooks:settings roles # skip when setup already turned it on
❯ /runbooks:list --review
❯ /runbooks:update billing.issue-invoice

❯ /runbooks:whats-next
❯ /runbooks:whats-next --since develop # the changes since develop

❯ /runbooks:orchestrator --goal "Implement all billing runbooks"
❯ /runbooks:orchestrator --goal "Make every draft runbook active" --max-agents 2

Where the files go

docs/runbooks/
├── runbooks.config.json  # what setup configured
├── roles.json            # roles, contexts and permissions, with role tracking on
├── README.md
├── billing/
│   └── issue-invoice.md  # one runbook, id billing.issue-invoice
├── _fixtures/            # responses of mocked runbooks
├── _contract/
│   └── openapi.json      # the API's OpenAPI document, for an API in another repository
├── _runs/                # one report per run (local)
└── _coverage/            # coverage reports (local)

Runbooks, fixtures, the contract, the role registry and the config are committed. Reports stay on your machine, because Git ignores them, and /runbooks:ui reads them from there. Versions before 1.3.0 also wrote runs.html and coverage.html to this folder. /runbooks:ui replaces them, and /runbooks:setup deletes them along with their lines in .gitignore. To change part of the config later, such as the test library or the viewports, run /runbooks:settings. /runbooks:upgrade updates the plugin, and offers to uninstall an install under the old name (runbook@fredsvanelli), so its commands stop showing next to the new ones.