---
# front-matter
---

# Runbook Template

Template credit goes to [CatieM20][]. If you haven't seen their talk on [Tackling Alert Fatigue][] yet, go give it a
watch. I've made a handful of modifications to include some site reliability engineering geared content.

[CatieM20]: https://github.com/CaitieM20/Talks/blob/master/TacklingAlertFatigue/runbook.md

## General

A quick description of the services. 1 to 2 sentences max. Why does this service matter? What is it's core
functionality? What Features does it provide users?

## Failure Mode and Effect Analysis

[FMEA][] is a method of failure analysis that helps teams create reliable systems and develop comprehensive on-call
response patterns.

[FMEA]: https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis

| Service   | Failure Mode         | Possible Cause  | Effects                           | Probability (P) | Severity (S)  | Detection (D) | Risk |
| :-------- | :------------------- | :-------------- | :-------------------------------- | :-------------- | :------------ | :------------ | :--- |
| DockerHub | Outage / Unreachable | DockerHub DDOSd | Cannot update or deploy extractor | remote (B)      | no effect (I) | high          | low  |

### Production Outage Scenarios

- [Example scenario](_outage.md)

## Dashboards

Links to the Dashboards for this service.

## Alerts

Links to the Alerts for this service

For Every Alert there should be a corresponding section in alphabetical order

### Alert Title

Alert Description: Why do we have this alert? What does it mean? What is typically the cause of this alert?

#### Impact to Customers:

How does this situation impact our customers? If the customers are not being impacted, this is a good indicator that the alert can be deleted.

#### Remediation Steps:

Checklist manifesto style steps for how to resolve this alert. A person who has never worked on our stack should be able to follow these steps and remediate the incident. If it cannot be remediated, include escalation steps here.

1.  Do this
2.  Check this graph
3.  Do this thing
4.  Do this other thing
5.  Verify service has recovered

## Deployment

How do you deploy this services. Favor Checklist manifesto style lists here as well.

1.  Do this thing
2.  Do this other thing
3.  Finally do this thing

### Canary Deploy

Instructions on how to do a Canary Deployment

1.  Do this canary thing
2.  another canary task

### Rollback Deploy

Instructions on how to Rollback a Deploy.

1.  Get the rollback build here
2.  Do this thing
3.  Do this other thing.