Runbooks: How to Write One That Works at 3am

A runbook is a written procedure for one specific, recurring operational situation, written so that someone who is not the expert can carry it out correctly under pressure. A good one names its trigger, tells the reader how to confirm the problem is real, gives exact commands rather than descriptions, and states when to stop and escalate.

Key takeaways

• A runbook covers one situation with a known response. If the response depends on judgement, what you need is a playbook.
• Write for the least experienced person who could plausibly be on call, not for yourself.
• Every line should be an action or a decision. Sentences explaining how the system works belong in documentation.
• State the exit criteria and a timed escalation. A runbook without an ending leaves the reader stuck at the bottom of the page at 4am.
• An untested runbook is a hypothesis. Put a last-verified date on it and treat a stale one as absent.

What a runbook actually is

The word is literal. Before it meant a document, a run book was the binder that sat beside a mainframe holding the operating instructions for the jobs that ran on it: what to start, in what order, and what to do when something failed. The modern meaning has not drifted far. A runbook is the written procedure for one recurring operational situation, aimed at whoever is holding the pager rather than at whoever built the system.

That definition carries a constraint most runbooks quietly break: one situation. A runbook covers a single trigger with a known response. The certificate expires in six days. The queue is backing up. The database has run out of connections. The log host is at 90% disk. If you catch yourself writing "if it is one of these five different problems", you are writing five runbooks and should split them, because the reader will be scanning rather than reading.

The second constraint is about the reader. Runbooks exist to close the gap between the people who understand a system and the people who happen to be awake when it breaks. If the only person who can execute a runbook is the person who wrote it, the document has achieved nothing, because that person could have been paged directly. The real measure is whether a competent engineer with no prior context on that system can follow it to a correct outcome.

Runbook, playbook, SOP: which one you are writing

These words get used interchangeably, and the distinction is worth keeping because it tells you how much structure to impose.

The practical difference between a runbook and a playbook is whether you already know the answer. If the correct response is known in advance and identical every time, write a runbook and push it as close to a script as prose allows. If the response depends on judgement, on what the graphs show, on who is available, then no amount of step numbering turns it into a runbook, and pretending otherwise produces a document that fails on first real use. Responding to a novel outage is a playbook. Restarting a stuck worker is a runbook.

The confusion runs in one more direction: a runbook is not documentation with numbered paragraphs. Documentation answers "how does this work". A runbook answers "what do I type". A reader at 3am does not have the working memory to derive the second from the first.

Why most runbooks fail

Runbooks fail in a small number of predictable ways, and nearly all of them come from writing for the wrong reader.

They describe the system instead of the actions

The most common failure by far. The author knows the architecture, so the author writes the architecture: the components, the data flow, the reasons. The reader wants a command. Anything in a runbook that is not an action, a decision, or context needed to make that decision is weight the reader has to carry past at the worst possible moment. Move it into documentation and link to it.

They assume the context the author has

"Restart the worker" is not a step. Which worker, on which host, with which command, as which user, and how does the reader know it came back? Every unstated detail is a point where the reader either guesses or stops. The bar is simple: the reader should never have to ask a question the runbook could have answered.

They have no decision points

Real procedures branch. If the queue is draining, wait. If it is not, restart. If restarting does not help, fail over. A runbook written as a flat list forces the reader to run every step, including the destructive ones, when the first might have been sufficient. Make the branches explicit and say exactly what to check in order to choose between them.

Nobody but the author has ever run them

A runbook that has never been followed end to end by a second person is a hypothesis. The failure modes are mundane and fatal: a command that relies on a shell alias only the author has, a permission the author holds and the on-call engineer does not, a hostname that changed two quarters ago.

They cannot be found under pressure

A perfect runbook in a wiki nobody can search is worth exactly as much as no runbook. The link belongs in the alert itself, so that the person woken by the page lands on the procedure instead of on a search box.

The anatomy of one that works

A working runbook has six parts, and only two of them are the steps themselves.

A header that lets the reader orient in ten seconds

• Trigger. The exact alert or symptom this responds to, worded the way the alert is worded so the two can be matched at a glance.
• Impact. One sentence on what users are experiencing. This is what the reader needs to judge urgency and to write the first status update.
• Severity and urgency. Whether this can wait until morning. Say it outright; the reader cannot infer it at 3am.
• Owner. A team, not a person. People change teams. The pager does not.
• Last verified. A date, and who last ran the whole thing end to end.
• Expected duration. Roughly how long this should take. A reader who knows it normally takes four minutes knows something is wrong at fifteen.

A verification step before anything else

The first instruction in every runbook should be a way to confirm the problem is real and still happening. Alerts fire on transient conditions, on monitoring faults, and on thresholds someone set optimistically two years ago. The cost of skipping this step is engineers performing recovery procedures on healthy systems, which is one of the more reliable ways to turn a minor alert into an actual outage.

Verification is also where the runbook says what it is not. If the check comes back clean, say so, and say what to do about it: close the alert, correct the threshold, or raise a ticket against the monitor itself.

Steps that are commands, not descriptions

Write the exact command, in a code block, with real values wherever it is safe to use them. Where a value has to be substituted, mark it unmistakably and say where to get it. Then say what correct output looks like, because a reader with no context cannot tell success from failure just by looking at it.

# BAD
Check the queue depth and restart the consumer if it is backed up.


# GOOD
1. Check queue depth:

     kubectl -n payments exec deploy/rabbitmq -- \
       rabbitmqctl list_queues name messages

   Normal is under 500 on payments.events. Above 5,000 means trouble.

2. If depth is above 5,000 AND FALLING, wait 5 minutes and re-check.
   Draining is expected after a deploy. Do not restart.

3. If depth is above 5,000 and NOT falling, restart the consumer:

     kubectl -n payments rollout restart deploy/payments-consumer

   Confirm with:

     kubectl -n payments rollout status deploy/payments-consumer

   Expect "successfully rolled out" within 90 seconds.

A rollback beside anything destructive

Any step that changes state needs its inverse written next to it, before the reader needs it. The moment someone discovers that a change made things worse is the worst available moment to start working out how to undo it.

Exit criteria

State how the reader knows they are done. Not "the service should recover", but the specific observation: error rate back under 1%, queue below 500, a real request to the endpoint returning 200. Without this the reader either stops too early or keeps intervening long past the point where the system had already recovered.

An escalation point with a clock on it

The last section of every runbook says what to do when the runbook does not work, and it should carry a time limit. "If the service has not recovered 20 minutes after step 4, page the platform on-call." A reader who has run out of steps and has no instruction will improvise, at the hour when improvisation is least reliable. Tie this to your existing escalation policy rather than inventing a second, competing one inside the document.

A complete example

Here is a short runbook with all six parts, for a situation almost every team has. It is deliberately unglamorous, because most runbooks are.

TRIGGER          Alert "cert-expiry-www" (threshold: 7 days remaining)
IMPACT           None yet. At expiry, every visitor sees a full-page
                 browser security warning and the site is effectively down.
SEVERITY         SEV-3 while days remain. SEV-1 the moment it expires.
URGENCY          Business hours if >72h left. Immediate if under 24h.
OWNER            Platform team
LAST VERIFIED    2026-08-11 by A. Novak
EXPECTED TIME    10 minutes


1. VERIFY the alert is real.

     echo | openssl s_client -servername www.example.com \
       -connect www.example.com:443 2>/dev/null \
       | openssl x509 -noout -enddate

   If notAfter is more than 7 days out, the alert is stale:
   close it and check the monitor's own clock. Stop here.

2. IDENTIFY who issues this certificate.

     kubectl -n prod get certificate www-tls \
       -o jsonpath='{.spec.issuerRef.name}{"\n"}'

   "letsencrypt-prod"  -> continue to step 3 (automated renewal).
   anything else       -> this cert is manual. Go to step 6.

3. CHECK why automatic renewal has not happened.

     kubectl -n prod describe certificate www-tls | tail -30

   Look at the Events block. The two common causes:
     - DNS-01 challenge failing   -> the _acme-challenge record is
                                     missing or stale. Fix DNS, then step 4.
     - Rate limited by Let's Encrypt -> wait. Do not retry in a loop.
                                     Re-check in 1 hour. If under 48h
                                     remain, go to step 6.

4. FORCE a renewal.

     kubectl -n prod delete certificaterequest \
       -l cert-manager.io/certificate-name=www-tls

   ROLLBACK: none needed. The existing certificate stays in place
   and serving until a new one is successfully issued.

5. CONFIRM (exit criteria). Within 5 minutes:

     kubectl -n prod get certificate www-tls

   READY must be True and NOT AFTER must be ~90 days out.
   Then re-run the step 1 command against the live endpoint and
   confirm the new notAfter date is being served. Close the alert.

6. ESCALATE.

   If READY is not True 20 minutes after step 4, or the certificate
   is manually issued, or under 24 hours remain at any point:
   page the platform on-call and post a status page notice under
   "Monitoring". Do not wait for expiry to communicate.

When a runbook should become code

Any runbook executed often enough, with no real decision points left in it, is a script waiting to be written. Make that progression deliberate rather than accidental, in three stages:

1. Manual. A human reads and types. Correct for anything rare, risky, or requiring judgement.
2. Assisted. The steps are wrapped in one command or a one-click job, but a human decides when to run it and watches it work. Most operational procedures should end their lives here.
3. Automated. The trigger fires the procedure with no human in the loop. Reserve this for procedures whose failure modes you understand well, and always with a cap on how many times it can fire before it wakes somebody.

Two cautions. Automating a procedure nobody has verified by hand means you now make the same mistake faster and unattended. And an automated procedure still needs its runbook, because when the automation fails, a person has to do by hand whatever it was doing.

Keeping runbooks from rotting

Runbooks decay faster than almost any other document, because the systems they describe keep changing while the document does not. A stale runbook is worse than no runbook, because a reader under pressure trusts it.

• Put a last-verified date on every runbook, and treat anything older than a year as unverified rather than merely old.
• Make it the rule that whoever uses a runbook fixes it. The five minutes just after an incident, while the discrepancy is still fresh, is the only moment anyone will willingly edit one.
• Review the runbook inside the postmortem whenever it was involved. "The runbook was wrong at step 3" is a finding, not a footnote.
• Delete runbooks for systems that no longer exist. An archive full of procedures for decommissioned services is what teaches people to stop trusting the whole collection.
• Exercise the important ones on purpose. One hour a quarter, in which someone who has never run a given runbook executes it against staging, finds more problems than a year of reading them.

Two numbers are worth tracking, because both are easy to count and both fall quietly when nobody is looking: the share of your paging alerts that have a runbook, and the share of those runbooks verified in the last six months.

Where runbooks meet monitoring

A runbook begins at a trigger, which means its usefulness is capped by the quality of the alert that fires it. Two failure modes live at that boundary and both are worth naming.

The first is alerts with no runbook attached. Every alert capable of waking someone should say what to do, and the link belongs in the alert body rather than somewhere the reader has to go hunting. This is the cheapest reliability improvement most teams have available to them: take an inventory of every alert that can page a human, and put a runbook link on each one. The alerts left without a link are usually the ones that should not have been paging anybody in the first place.

The second is runbooks whose verification step keeps coming back clean. If step 1 regularly says "the problem is not real", the runbook is not the thing that needs fixing. A check that alerts on a single failed request from a single location will wake somebody for a blip that never touched a user. Confirming from more than one location before alerting, and requiring more than one consecutive failure, removes most of that noise, which is worth more to an on-call rotation than any amount of procedural polish.

The reverse gap is worth an inventory too: incidents your customers noticed and your alerts did not. Those are usually not outages but service degradation, where the system answers every request and simply does so slowly or incompletely. A runbook cannot help with a situation that never triggers a page.

See how Pulsetic's website monitoring catches this from the outside, across 15+ locations.