plugin · v1.18.1
■ runbooks
Spec-Driven User Acceptance Test. UAT runbooks kept as a project's verified knowledge base: write, run in the browser or through API test scripts, audit against the code, measure coverage.
Why use Runbooks?
An end-to-end suite tells you whether the code still does what the tests say. It can't tell you whether the tests say the right thing, which flows they skip, or whether the agent that wrote the feature checked anything at all. Most teams find these gaps in production. Runbooks close them before the release.
The problems
Nobody reads the test suite
A Playwright spec is selectors, waits and fixtures. A product manager can't review it against the ticket, and a new developer can't learn the product from it. So the knowledge of how the product should behave lives in people's heads and in old tickets, and it leaves when they do.
The agent grades its own work
When an agent writes a feature and its tests in the same session, both carry the same misreading of the ticket. The tests assert what the code does, so they pass when the code does the wrong thing. And an agent asked to verify a flow often queries the database or calls the endpoint with curl, then reports success for a flow no user could finish in the browser.
You can't see what isn't tested
A test report covers the tests that exist. It says nothing about the routes and endpoints no test touches, and the next bug ships from one of those.
Tests rot without telling you
A renamed button breaks a selector. Someone marks the test skip to unblock a release. Six months later the suite is green and checks half of what it used to. Nothing in the suite tells you which tests still match the product.
One bug turns forty tests red
Each end-to-end test logs in, creates its data and walks its flow from the start. When the login page breaks, every test fails, and you open forty traces to find one cause.
Testing waits for test code
A flow gets checked only after someone writes its spec, its fixtures and its page objects. On a new feature that work comes last, so the first real check is often a person clicking through it the day before the release.
The DOM passes and the screen fails
An assertion on text and selectors passes while the layout breaks on a phone, or while the page drifts away from the design everyone approved.
The solutions runbooks provide
A spec anyone can review
Compare a test for issuing an invoice with the runbook step for the same flow. The test passes as long as the toast contains the word "issued", even if the invoice went to the wrong client. Few reviewers would catch that in the test code.
test("admin issues an invoice", async ({ page }) => {
await page.goto("/billing/invoices");
await page.getByRole("button", { name: "New invoice" }).click();
await page.locator("#client-select").selectOption("acme");
await page.locator("[data-testid=amount]").fill("1200");
await page.getByRole("button", { name: "Issue" }).click();
await expect(page.locator(".toast")).toContainText("issued");
});### S02. Admin issues an invoice to a client
Parents: S01
Page: `/billing/invoices`
Actions:
1. Click "New invoice".
2. Pick "Acme" in "Client" and type 1200 in "Amount".
3. Click "Issue".
Expected: the toast "Invoice issued", and the row "Acme | $1,200.00 | Sent" at the top of the table.The runbook names each control by the label a user sees and states the exact text the page must show. A product manager reads it in the pull request, next to the code, and spots a wrong amount or a missing status. The file lives in docs/runbooks/, so it is still there when the people who wrote the feature move on.
The agent has to prove it
Write the runbook from the ticket or the spec with /runbooks:create --from ticket, review its Expected lines, and the feature has to meet them. The agent can't take a shortcut to a pass. It reads no database, writes no SQL and confirms nothing through a direct HTTP call. Only the browser decides a ui step, and only the test script decides an api step. A pass means a user could do what the step describes.
Setup adds a block to your AGENTS.md or CLAUDE.md. From then on, feature work creates or updates the runbook of every flow it changes, and before the agent calls the work done it runs /runbooks:run --affected. A failing runbook means the work is not done.
Coverage you can see
/runbooks:map lists every route and endpoint no runbook cites and writes a placeholder for each flow, so the gaps show up before you write a single runbook. /runbooks:coverage counts a route as covered only when an active runbook cites it, which means a runbook that passed a full run. It then suggests up to five runbooks to write next, riskiest flows first.
Drift shows up the day it happens
/runbooks:audit reads each runbook against the code. It flags a runbook as broken when it cites a route or step that no longer exists, and as drift when a label, an expected text or a request field no longer matches. --fix applies the edits. You find out which runbooks went stale without waiting for one to fail.
One bug, one failure
Each step lists the steps it depends on. When a parent fails, the agent skips the branch below it and marks those steps blocked-by-parent, so a broken login shows up as one failure with one cause. The agent also runs every step before it fixes anything, so a single run lists every failure in the flow. See steps that depend on steps.
A check on day one, a script when it pays off
A ui runbook runs in the browser as soon as it exists, with no test code. The agent drives Playwright through its MCP server, or you drive and report each result in guided mode. When the flow settles, /runbooks:to-test turns the runbook into a test in your library (Playwright, Cypress, Jest, Vitest, PHPUnit, Pest) or adopts the test that already covers it. You write test code for the flows worth automating, and every other flow still gets checked.
Screens checked against the design
A ui runbook runs once per screen size it lists, such as desktop and then mobile. On each pass the agent compares every linked screen with its design in Figma, Claude Design, Adobe XD or local files, and the report lists each difference as major or minor. The Expected line still decides the step, so a design note never turns a pass into a failure.
Runbooks and your end-to-end tests
Runbooks don't ask you to delete your suite. map notes the existing tests that fit each flow, to-test adopts them, and an api runbook runs through a test script from the start. The runbook records what must happen. The test is one fast way to check it in CI. Without the runbook, the test is the only record of the behavior, and only developers can read it.
| Question | E2E tests alone | With runbooks |
|---|---|---|
| Who can review what gets tested | Developers who read the test code | Anyone who reads Markdown |
| Where expected results come from | Whatever the test author asserted | Expected lines reviewed in the pull request |
| Flows nothing tests | Invisible | Listed by map, counted by coverage |
| Tests that no longer match the code | Found when they break, or never | Flagged by audit as broken or drift |
| A broken first step | Every test after it fails | One failure, the branch below marked blocked-by-parent |
| The first check of a new flow | After someone writes the test | As soon as the runbook exists |
| Screen sizes and the design | Unchecked unless you add visual tests | Each linked screen compared with its design, per viewport |
| When an agent calls work done | When it says so | When the runbooks the change touched pass |
Start with the gaps
See how it works. Install the plugin, then run /runbooks:setup and /runbooks:map. map writes a placeholder for every flow nothing covers, and /runbooks:coverage turns that list into a number you can track from the first day.