Why do we use the term “root cause”? I especially love the opening analogy.

  John Allspaw — The ReadME Project

Our Reliability Manifesto is a succinct collection of rules, guidelines, and best practices that reflect our current thinking on what it takes to build a reliable system.

  Christian Hardenberg — Delivery Hero

Here’s how New Relic sets their S*Os.

Set SLIs and SLOs against system boundaries

  Dan Holloran and Elisa Binette — New Relic

It involves lots of machine learning and a “team resilience score”.

  Jennifer Riggins — The New Stack

Every incident is unique, so incident analysis is about learning in order to improve resilience, rather than trying to “fix” a “root cause”.

  Laura Maguire — Jeli

A lot of incident management guides out there are aimed at established, big-scale companies. Things are different when you’re in startup mode.

  Chris Evans — incident.io

This is so cool! It’s a guide for what kinds of incidents you’re likely to learn the most from. There’s a long list of things to look out for with explanations.

  Laura Maguire and Vanessa Huerta Granda — Jeli

Oh, now this is fascinating. Firefox, like, the web browser itself, had an outage in January. It just stopped working for everyone.

  Christian Holler — Mozilla

If you’re looking for an explainer on the CAP theorem, this article gives a great overview with practical details.

  Bartłomiej Żyliński — SoftwareMill

This is about what the security field can learn from SRE. Obviously not directly applicable to SRE, but this article gives us a really great outside perspective.

  Anton Chuvakin — Security Boulevard

Code doesn’t “rot”, but the environment around it changes constantly.

  Lorin Hochstein

How do complex service dependencies affect your SLA? What if service A depends on service B and C being up, but service D or E being up?

TLDR; for serial, multiply availability; For parallels, multiply unavailability.

  Alex Ewerlöf

Here’s how and why not to be a hero. It’s bad for you and everyone else too.

  Isaac Seymour — incident.io

This followup to their initial incident report has a lot to learn from, especially if you run Consul at scale.

  Daniel Sturman and others — Roblox

This week, I came across the Byford Dolphin diving bell incident. This accident seems at face value to be “human error”, but there’s so much to it. Content warning: the accident was quite grisly.

  Wikipedia

Canary testing is more than just deploying your code to a small part of your fleet. You need a plan for how you’re going to spot problems.

  Jyoti Sahoo — OpsMx

My favorite part is how they look for changes in performance, rather than using a static threshold.

  Angus Croll — Netflix

It pays to think ahead about how you’ll answer questions from execs during an incident.

  Chris Fenning — DZone

On January 24, 2022, as a result of an internal Cloudflare product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin.

  Jeremy Hartman — Cloudflare

An analysis of SRE job descriptions from 4 companies highlights what businesses actually expect SREs to do.

  JP Cheung — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Members of the search giant’s site reliability group say managers fostered a toxic environment. Google says a ‘safe, inclusive workplace’ is a top priority.

  Nico Grant — Bloomberg

In the past, NASA has increased the likelihood of mission success by sending duplicate spacecraft. In the case of the JWST, that’s not an option.

  Robert Barron

This article makes a case that agile development practices depend on SRE.

  Ash P — Cruform Newsletter

This history covers the advent of the Incident Command System (ICS) and subsequently the National Incident Management System (NIMS).

  JJ Tang — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Meta migrated their Facebook Ordered Queueing Service (FOQS) system to a global, highly-available deployment. This article describes the original architecture, lists its shortcomings, and explains how they did the migration with zero downtime.

  Jasmit Kaur Saluja and Dillon George — Meta

This is the first time I’ve heard of a “Problem Manager” role, and I like it.

  Laurel Frazier — Transposit

How do you make an SLO for a service with long-running requests? One method is to report metrics on regular time intervals.

  Liz Fong-Jones — Honeycomb

A failure in their Software-Defined Networking (SDN) configuration system required manual recovery.

  Google

[…] when Kubernetes is involved, the number of alert sources can skyrocket quickly. This article will reflect on some common causes of alert fatigue and share tips to help reduce it.

  Nate Matherson — DZone

Meta has a special system to warn servers about power outages, giving them 45 seconds of battery power to finish things up and get ready to shut down.

  Raghunathan Modoor Jagannathan, Sulav Malla, and Parimala Kondety — Meta

This is an approachable explanation of the Paxos algorithm with examples, diagrams, and code.

  Martin Fowler

But what does reliability mean for people outside of engineering? And how does it translate into best practices for other teams?

  Emily Arnott — Blameless

“The Practice of Practice” is a concept from improvisational music. This article artfully applies the idea to the practice of incident response.

  Matt Davis — Blameless

I haven’t heard of this technique being used before, assigning alerts to on-call folks in round-robin order as they come in. I wonder if there’s a reason for that…

  Hannah Culver — PagerDuty

Raise your hand if you’ve been bitten by DNS before.

  Julia Evans