When everything is down: a simple incident playbook

An outage on a small team usually goes like this. Someone notices the site won’t load and posts in the general chat. Three people log into the server at once, and one restarts the database while another reads its logs. Customers write to support, and support has no idea what to tell them. An hour later everything works again, and nobody can quite say what happened.

That isn’t a lack of professionalism. Under stress people act on habit, and there’s no habit for this. A one-page plan written on a calm day stands in for one.

What to write down in advance

Who can declare an incident. Answer: anyone. Declaring and then cancelling is cheaper than spending half an hour debating whether this counts yet.
Where the conversation happens. A dedicated channel per incident, not the general chat.
Who has production access and where the credentials live.
How to reach support at your host, database provider, payment provider and email service.
Where updates for users go: a status page, an in-app banner, an email.

Roles

Even if there are only two of you, split the roles:

The lead makes decisions, keeps track of time, and doesn’t fix things hands-on. Their job is to see the whole picture.
The fixer investigates and makes changes, telling the lead what they’re doing.
The communicator writes to users and answers support, so nobody interrupts the fixer.

On a team of two, the lead and the communicator are the same person. What matters is that the lead and the fixer are different people. Someone deep in logs won’t notice that forty minutes have passed.

The sequence

Declare the incident and open the channel. The lead notes the time.
Establish scope: what’s broken, for everyone or some people, since when.
Stabilise first, find the cause later. If something shipped recently, roll it back. If the new feature has a flag, turn it off. The root cause can wait for a working system.
Keep a timeline in the channel: what was noticed, what was tried, what changed. A week later it’s the only reliable record.
Update users at regular intervals, even when there’s no news.
Confirm that everything has actually recovered, and only then close the incident.

How to write to users

People fill silence during an outage on their own, usually with the worst explanation available. A short message right away beats a detailed one an hour later. It needs three things: what’s broken, what still works, and when the next update is coming.

Subscription payments are failing right now. Login and project
work are available, and your data is safe. We're on it; next
update at 15:30.

Don’t promise a fix time you don’t know. Promise the time of the next update.

A review without blame

A few days later, while it’s fresh, hold a short review. The goal is to understand how the system allowed this to happen, not who pressed the button. If one person could cause it with one action, the problem is that the action had no safeguard. People who get singled out in reviews start keeping quiet about their mistakes, and outages get longer.

The written review covers:

what happened and how long it lasted;
who was affected and how;
a timeline built from the channel;
what helped and what got in the way;
two or three actions with owners and dates.

The last item matters most. A long list of improvements where none get done is worse than a short one that’s done in full.

Drills

A plan nobody has opened won’t be remembered when it counts. Every few months, stage an outage on staging: switch off the database or break the config, and walk the plan from start to finish. You’ll usually find a couple of places where it doesn’t match reality.

SUBXCAT can help write a plan like this for your product and run the first drill with your team.