Even when your system has redundancy, sometimes all the redundant copies fail at once because of what they share in common.

  Marc Brooker

Feature flags make it easy to roll out database schema migrations without downtime. This example uses double-writing and a data migration script.

  Tom Hombergs — Reflectoring

Like some kind of Netflix of SRE writing, incident.io just dropped an entire guide on incident management, ready for bingeing. My favorite is the section on on-call compensation.

  Chris Evans — incident.io

A major part of SRE is deciding what level of reliability makes sense, and how prepared you should be. This article drives that point home with an analogy to the James Webb Space Telescope.

  Robert Barron — IBM

Ably posted this design overview of their HA real-time messaging system, with lots of juicy details.

  Jo Stichbury — Ably

An advice columnist helps a newbie on-caller ease into the pager life.

  Liz Fong-Jones — Honeycomb

I like that this article advocates using different templates for different kinds of retrospectives with different goals.

  Myra Nizami — Blameless

Yes, we need more of this! The skills covered are: Communication, Empathy, Teamwork, Motivation, and Documentation.

  Paul Marsicovetere — Formidable

Catchpoint and Blameless have teamed up on this year’s SRE survey. They’ve sweetened the deal with two $5 donations to charity for every survey completed. Go do it!

  Kurt Andersen — Blameless

I sure miss the good old “checkmark-i” icon. Oh wait, no I don’t.

  Jeff Martens — Metrist

How can you handle failure gracefully? Click through for 6 strategies to consider.

  Boris Cherkasky — Riskified

Declaring the first incident when you start a new job can be intimidating, but it really shouldn’t be. Let’s look at some common fears, and work out how to address them.

  Isaac Seymour — incident.io

The incident involved fiber equipment failure and a suboptimal automated remediation.

  Google

This is a primer on Urgency and Impact in incidents, including the difference between them and how to use them.

  Noor-ul-Anam Ruqayya — Blameless

Running retrospectives on near-miss incidents can be highly valuable, as this article discusses.

  Vanessa Huerta Granda — Jeli

This article really upends the concept of “human error”, in an intriguing way.

  Lorin Hochstein

A key part of building reliable systems is often overlooked: continuously learning.

In the highly dynamic CI/CD environment, engineers with stale or outdated knowledge of the system are less able to detect, diagnose or repair anomalous behavior in their systems […]

  Laura Maguire (jeli.io) — The New Stack

This is the story of how an organization transitioned from a single NOC-like on-call team to individualized alerts routed to the relevant team.

  Boris Cherkasky

This guide has a set of key factors you should consider when building a new SRE team in order to increase the likelihood of success.

  Ash P — SREPath

My favorite pitfall discussed in this article: avoid committing to every possible remediation action from every incident.

  incident.io

This article, written by one of the authors of the Google SRE book, is a critical look at the state of SRE and what the future holds.

Today, I believe we cannot successfully answer several key questions about SRE.

  Niall Murphy

This interview goes into the thorny challenges around building a reliable app based on third-party services. It delves into the lack of reliable reporting we commonly see from cloud service providers and what ideal reporting would look like.

  Jeff Martens (interviewing Jeff Smith) — Metrist

We’ll start off this week with a recap of a KubeCon talk that urges leaving the concept of “human error” behind.

  Jennifer Riggins — The New Stack
  Talk by Silvia Pina

Just to be clear, they’re saying the tips are written by Instacart’s first SRE — they’re not tips aimed oddly specifically at the second Instacart SRE. Good tips, too.

  Quentin Rousseau — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

This is a really good point, and well argued. Then there’s an amusing bit at the end about alerting on the number of WARNING-level log messages generated by the system as a proxy for overall health.

  Chris Siebenmann

In this post, I’m going to expand on the values we’re currently using at Honeycomb to monitor on-call health, why we think they’re good, and some of the challenges we’re still encountering.

  Fred Hebert — Honeycomb

Internal and external communication are critical in an incident, second (perhaps) only to actually resolving the problem. Read this article to learn about who you need to communicate with, how to talk to them, and how to prepare in advance.

  Hannah Culver — PagerDuty

If you’re playing the hero role at your organization, you might be unintentionally masking the need for better incident management practices.

  Malcolm Preston — FireHydrant

I chatted with Emily Arnott of Blameless for a solid hour about everything from the origins of this newsletter and how I make it, to my thoughts on SRE and where it’s going. Somehow she managed to fit it all into this article. Thanks, Emily!

  Emily Arnott — Blameless

The section on TTR (Time To Recovery) really caught my eye, both by confirming that MTTR is generally not a useful metric, and also finding one case where TTR does seem to be predictive.

The Spotify engineering blog seems to be down as of this publishing, so here’s the archive.org version.

  Clint Byrum — Spotify

SRE concepts apply wonderfully well to compliance and governance. Each field has a lot to learn from the other.

  Jennifer Riggins — The New Stack

More than ever, we should all be focused on shipping great products, retaining high-demand engineers, and building trust with customers. And investing in a thoughtful incident management strategy is one way to get there. Let’s explore how.

  Robert Ross — FireHydrant

At this week’s DevOps Enterprise Summit (DOES) Europe, Vanguard talked about how they made the move from traditional architecture to the majority in the cloud, adopted site reliability engineering and even built their own customer-facing SaaS.

  Jennifer Riggins — The New Stack

This article has a great discussion of the risks of larger, less frequent deploys. It goes on to explain how they transitioned to smaller and more frequent deploys while focusing on safety.

  Will Sewell — Monzo

What makes this article special is its focus on addressing the common concerns that people have when you try to get them to own their code for its full lifecycle. It offers practical advice to win folks over.

  Martha Lambert — incident.io

Sounds like there were some pretty great talks at SRECon. I gotta admit, I’m kinda having some FOMO.

  Emily Arnott — Blameless