What should the first line of a runbook say?

It should say how to stop the damage right now, before any description of what the system does or why it exists. The person reading it during an incident needs the mitigation first (turn off the flag, route to the old path, scale the worker to zero), and the explanation is only useful once the damage has stopped.

Why do most runbooks fail during a real incident?

They are written for a calm reader with time, and the actual reader is tired, under pressure, and has minutes. Anything that requires reading three paragraphs to find one command, or that assumes context the reader has lost at three in the morning, will be skipped in favour of guessing.

What should be cut from a runbook?

Architecture description, project history, rationale for past decisions, and any step phrased as "investigate the root cause". Those belong in design documents, because during an incident they cost reading time and give nothing back, and their presence makes the useful parts harder to find.

How do you know whether a runbook is any good?

Give it to someone who has never worked on that system and ask them to follow it on a staging failure while you stay silent. Every question they ask out loud is a defect in the document, and this test finds more problems in twenty minutes than a year of review comments.

Should runbook commands be copy-pasteable?

Yes, and this is the single most valuable formatting rule. A command with a placeholder the reader has to fill in from memory is a command that gets typed wrong under pressure, so give the exact form with a clearly marked variable and say where the value comes from.

How do you keep runbooks from going out of date?

Attach them to the alert that fires, review them in the incident write-up whenever one is used, and delete any runbook for a system that no longer exists. An out-of-date runbook is more dangerous than a missing one, because someone will follow it and take an action that no longer applies.

What does a runbook need to say about verifying the fix?

It needs to name the specific signal that shows the action worked and roughly how long it takes to appear. Without that, the responder runs the command and then stares at a dashboard unsure whether to wait, retry, or escalate, which is where the second mistake in an incident usually happens.

Why do generated codebases need runbooks more than hand-written ones?

Because the informal knowledge that is normally held in one engineer's memory was never formed. On a hand-written system somebody remembers why the retry limit is three, and on a largely generated system that memory does not exist anywhere, so the operational document becomes the only record of how it behaves under stress.

How long should a runbook be?

Short enough that the mitigation is visible without scrolling, which usually means one screen for the emergency section and a longer reference section below it. Length itself is not the problem, but putting the action that stops customer harm below a lot of context is.

Who should write the runbook for a feature?

Whoever built it, before it reaches production, and it should be reviewed by someone who was not involved. The builder knows the failure modes, and the outside reviewer is the only person who can tell which assumed knowledge is missing from the page.

What belongs in the runbook about waking other people?

Names, what each person actually knows, and the threshold that justifies calling them. A responder who is unsure whether the situation warrants waking someone will usually wait too long, so the document should remove that decision by stating the condition explicitly.

What belongs in a runbook nobody reads: Reveneau

The runbook that gets written and the runbook that gets used are different documents, and most teams only write the first one.

The written one describes the system. It has an architecture summary, a paragraph about why the queue was chosen, a diagram, and a numbered list of diagnostic steps that begins with "check the logs". It reads well. Somebody approved it in a pull request. It is a reasonable piece of work.

The used one gets opened at 3:14 in the morning by an engineer who was asleep nine minutes ago, on a phone hotspot, while a customer-facing page returns errors and a manager is typing "any update?" into a channel. That reader has about ninety seconds of usable attention before they abandon the document and start guessing.

Nearly nothing in the first document is useful to the second reader. So write for the second one.

Put the mitigation in the first line

Before the description. Before the diagram. Before any sentence that begins "this service is responsible for".

The first thing on the page should be how to make the damage stop, stated as an action. Turn off the feature flag. Route traffic back to the previous version. Scale the worker to zero. Disable the integration. Whatever the simplest fix is, name it, and say what it costs to use it, because the responder needs to know whether using that fix loses data or just loses a feature.

This ordering feels wrong to engineers, and I understand why. It puts the crude fix above the correct one, and it feels like encouraging people to act without understanding. But the responder is going to act without full understanding regardless, because that is what an incident is. The choice is whether they act on the documented simple fix or on their own improvised guess at half past three.

Understanding comes second. Stopping the damage comes first.

Give exact commands, not descriptions of commands

"Check the queue depth" is not a step. It is a task assigned to someone who is already overloaded.

The step is the command, in a code block, with the real host name and the real flags, ready to paste. If part of it varies, mark that part clearly and say exactly where the value comes from and what a normal value looks like. If a command needs credentials or a VPN, say so before the command rather than after the reader has tried it and got a permission error.

This is unexciting work and it is where most of the value is. Under pressure people retype commands wrong, misremember flag names, and pick the wrong environment. A runbook that removes typing removes a whole class of mistakes, including the worst one, which is running the right command against production when you meant staging.

Say how to know it worked

This is the step almost every runbook omits, and its absence causes the second mistake in most incidents.

The responder runs the command. Then what? Which graph should move, in which direction, and within how long? Is thirty seconds of no change normal, or does it mean the action failed? If nothing has changed after two minutes, what is the next thing to try?

Without that, you get an engineer staring at a dashboard, unsure whether to wait, run it again, or escalate. Running it again is often harmful. So write the confirmation signal next to every action: "error rate on the checkout dashboard should fall within about sixty seconds; if it has not moved after two minutes, the flag did not propagate and you should do X instead."

Name the people, and name the threshold

Every runbook should say who to wake and what each of them actually knows. Not a role, a person, with the specific area: "Priya knows the payment reconciliation job. Marcus set up the queue configuration."

Then give the condition that justifies waking them, in plain terms. Responders left to judge this themselves almost always wait too long, because nobody wants to be the person who woke a colleague over something small. If the document says "if customer data may have been written incorrectly, wake someone immediately regardless of the hour", the decision is already made and the responder is free to act.

What to cut

Architecture explanations. Design history. The reasoning behind the retry policy. Anything written to satisfy a reviewer rather than a responder. All of that is genuinely useful and belongs in a design document that someone reads on a Tuesday afternoon.

Also cut any step that says "investigate the root cause". That is a description of the entire job, written as if it were an instruction. If you know the likely causes, list them as specific checks with specific commands. If you do not, say honestly that this failure mode has not been seen before and here is who to call.

The reason to cut is not tidiness. Every extra paragraph pushes the useful line further from the top of the page, and position on the page is what matters most for a reader who is scanning rather than reading.

Test it the only way that works

Hand the runbook to someone who has never worked on that system. Break the thing in staging. Ask them to follow the document while you sit there and say nothing.

Every question they ask out loud is a defect. "Where do I find that dashboard?" is a defect. "Which environment is this?" is a defect. "Is this the same as the other queue?" is a defect. Twenty minutes of this finds more problems than a year of review comments, because a reviewer reads the document and a responder uses it, and those produce completely different failure lists.

Staying silent is the hard part. Every instinct will tell you to help.

Why this matters more on generated systems

Here is what changed, and it is the reason we keep returning to this topic.

A hand-written system depends on a lot of undocumented operational knowledge held in people's memory. Someone remembers that the retry limit is three because of a rate limit at a payment provider in 2024. Someone knows the nightly job is slow on the first of the month. That knowledge is easily lost and held by few people, and it is real.

On a largely generated system, that memory was never formed. Nobody sat with the retry logic long enough to build an intuition about it. The 2025 DORA report, drawn from around 5,000 respondents, found AI adoption relating positively to throughput and negatively to delivery stability, with the direct summary that "AI doesn't fix a team; it amplifies what's already there". A team whose operational knowledge was already held by one person, producing three times the change, now has three times as much to know and still only one person who knows it.

So the runbook stops being a convenience for whoever is on call. It becomes the only place the knowledge exists. Write it while the feature is being built, when someone still remembers what the failure modes were, rather than after the first incident teaches them.

The same argument appears in handing over AI-written software, and the wider version of the problem is in how to take over a codebase you did not write.

The part about going out of date

A runbook describing a system that changed six months ago is worse than nothing, because somebody will follow it. They will run a command against a service that was renamed, or disable a flag that no longer controls anything, and lose four minutes they did not have.

Three habits keep them accurate. Link the runbook directly from the alert that fires, so it is opened often enough for errors to appear. Review it in the write-up every time an incident uses it, while the gaps are still remembered. And delete runbooks for systems that no longer exist, without discussion.

Do not try to keep a large library current. Keep a small one current instead. Coverage of everything, accurate about nothing, is the state most documentation ends in, and it is worse than a short collection you trust. The related discipline for the checks themselves is in when evals give false confidence, which is the same failure in a different form: a document or a test suite that passes because nobody has questioned it.

Thanks to the on-call engineers who told us which runbooks they gave up on and at what point. It was almost always the same point, about four paragraphs in, when it became clear the document was going to explain the system before it told them what to do.

A runbook is an instruction to a frightened person, and it should read like one.

Related guide: Handing over AI-written software.

Sources

2025 DORA report announcement, Google Cloud: around 5,000 respondents; AI adoption relates positively to throughput and negatively to delivery stability, and amplifies a team's existing strengths and weaknesses.

What belongs in a runbook nobody reads