Hi folks! I’m taking a sick day today, so no issue this week.
A primer on what makes a good runbook.
Runbooks are most effective when they are readily available, easily actionable, and up-to-date and accurate.
In this article, we describe the architecture and implementation of our SRE infrastructure, how it is used and how it was adopted.
Philipp Gündisch and Vladyslav Ukis — Siemens
After an explanation of tech debt, this article goes into a possible solution: having on-call folks fix lingering problems in between pages.
Dormain Drewitz — The New Stack
I’ve read plenty of articles about service ownership, but this one has something new: a discussion of how to divvy up a monolith into separate “services” for teams to own.
Hannah Culver — PagerDuty
The folks at Sendinblue have chronicled their journey to better incident response, and there’s a lot here to learn from.
Tanguy Antoine — Sendinblue
Incidents will always happen, but thankfully they have plenty of upsides, as this article explains.
Andre King — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.
You’re not getting paged. Is it because you’ve fixed all the things, or has your alerting atrophied?
The folks at incident.io are here with the results of their survey of on-call practices. I like the focus on compensation for being on-call.
- Netflix briefly went down for some users after the new ‘Stranger Things’ episodes debuted Friday morning, according to outage reports
I noticed this one while trying to read one of their articles. I was getting NXDOMAIN trying to resolve zebrium.com.
Less than 12 hours after their outage, Cloudflare posted this detailed run-down of what happened.
Tom Strick and Jeremy Hartman — Cloudflare
Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed.
By “derivatives”, the author means rate-of-change, like Prometheus’s
irate(). Derivatives have their place, but this article has good reasons to reconsider using them for alerts.
In this article, I’ll dive into what it takes to get into site reliability engineering, how to adopt it within your own organization and some of the core principles and best practices you’ll need to keep in mind as you move forward in your SRE maturity journey.
Jemiah Sius — devops.com
I have given and received this question in many SRE interviews, and it’s famously used by Google in their interviews. This article dissects the question and its merits and downsides for the benefit of both interviewers and interviewees.
Cloudflare had a major outage, taking many sites and services with it.
Even when your system has redundancy, sometimes all the redundant copies fail at once because of what they share in common.
Feature flags make it easy to roll out database schema migrations without downtime. This example uses double-writing and a data migration script.
Tom Hombergs — Reflectoring
Like some kind of Netflix of SRE writing, incident.io just dropped an entire guide on incident management, ready for bingeing. My favorite is the section on on-call compensation.
Chris Evans — incident.io
A major part of SRE is deciding what level of reliability makes sense, and how prepared you should be. This article drives that point home with an analogy to the James Webb Space Telescope.
Robert Barron — IBM
Ably posted this design overview of their HA real-time messaging system, with lots of juicy details.
Jo Stichbury — Ably
An advice columnist helps a newbie on-caller ease into the pager life.
Liz Fong-Jones — Honeycomb
I like that this article advocates using different templates for different kinds of retrospectives with different goals.
Myra Nizami — Blameless
Yes, we need more of this! The skills covered are: Communication, Empathy, Teamwork, Motivation, and Documentation.
Paul Marsicovetere — Formidable
Catchpoint and Blameless have teamed up on this year’s SRE survey. They’ve sweetened the deal with two $5 donations to charity for every survey completed. Go do it!
Kurt Andersen — Blameless
I sure miss the good old “checkmark-i” icon. Oh wait, no I don’t.
Jeff Martens — Metrist
How can you handle failure gracefully? Click through for 6 strategies to consider.
Boris Cherkasky — Riskified
Declaring the first incident when you start a new job can be intimidating, but it really shouldn’t be. Let’s look at some common fears, and work out how to address them.
Isaac Seymour — incident.io
The incident involved fiber equipment failure and a suboptimal automated remediation.
This is a primer on Urgency and Impact in incidents, including the difference between them and how to use them.
Noor-ul-Anam Ruqayya — Blameless
Running retrospectives on near-miss incidents can be highly valuable, as this article discusses.
Vanessa Huerta Granda — Jeli