Runbook Template

Template credit goes to CatieM20. If you haven't seen their talk on [Tackling Alert Fatigue][] yet, go give it a watch. I've made a handful of modifications to include some site reliability engineering geared content.

General

A quick description of the services. 1 to 2 sentences max. Why does this service matter? What is it's core functionality? What Features does it provide users?

Failure Mode and Effect Analysis

FMEA is a method of failure analysis that helps teams create reliable systems and develop comprehensive on-call response patterns.

Service	Failure Mode	Possible Cause	Effects	Probability (P)	Severity (S)	Detection (D)	Risk
DockerHub	Outage / Unreachable	DockerHub DDOSd	Cannot update or deploy extractor	remote (B)	no effect (I)	high	low

Production Outage Scenarios

Example scenario

Dashboards

Links to the Dashboards for this service.

Alerts

Links to the Alerts for this service

For Every Alert there should be a corresponding section in alphabetical order

Alert Title

Alert Description: Why do we have this alert? What does it mean? What is typically the cause of this alert?

Impact to Customers:

How does this situation impact our customers? If the customers are not being impacted, this is a good indicator that the alert can be deleted.

Remediation Steps:

Checklist manifesto style steps for how to resolve this alert. A person who has never worked on our stack should be able to follow these steps and remediate the incident. If it cannot be remediated, include escalation steps here.

Do this
Check this graph
Do this thing
Do this other thing
Verify service has recovered

Deployment

How do you deploy this services. Favor Checklist manifesto style lists here as well.

Do this thing
Do this other thing
Finally do this thing

Canary Deploy

Instructions on how to do a Canary Deployment

Do this canary thing
another canary task

Rollback Deploy

Instructions on how to Rollback a Deploy.

Get the rollback build here
Do this thing
Do this other thing.

2.5 KiB Raw Blame History