This followup to their initial incident report has a lot to learn from, especially if you run Consul at scale.

  Daniel Sturman and others — Roblox

This week, I came across the Byford Dolphin diving bell incident. This accident seems at face value to be “human error”, but there’s so much to it. Content warning: the accident was quite grisly.

  Wikipedia

Canary testing is more than just deploying your code to a small part of your fleet. You need a plan for how you’re going to spot problems.

  Jyoti Sahoo — OpsMx

My favorite part is how they look for changes in performance, rather than using a static threshold.

  Angus Croll — Netflix

It pays to think ahead about how you’ll answer questions from execs during an incident.

  Chris Fenning — DZone

On January 24, 2022, as a result of an internal Cloudflare product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin.

  Jeremy Hartman — Cloudflare

An analysis of SRE job descriptions from 4 companies highlights what businesses actually expect SREs to do.

  JP Cheung — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Members of the search giant’s site reliability group say managers fostered a toxic environment. Google says a ‘safe, inclusive workplace’ is a top priority.

  Nico Grant — Bloomberg

In the past, NASA has increased the likelihood of mission success by sending duplicate spacecraft. In the case of the JWST, that’s not an option.

  Robert Barron

This article makes a case that agile development practices depend on SRE.

  Ash P — Cruform Newsletter

This history covers the advent of the Incident Command System (ICS) and subsequently the National Incident Management System (NIMS).

  JJ Tang — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Meta migrated their Facebook Ordered Queueing Service (FOQS) system to a global, highly-available deployment. This article describes the original architecture, lists its shortcomings, and explains how they did the migration with zero downtime.

  Jasmit Kaur Saluja and Dillon George — Meta

This is the first time I’ve heard of a “Problem Manager” role, and I like it.

  Laurel Frazier — Transposit

How do you make an SLO for a service with long-running requests? One method is to report metrics on regular time intervals.

  Liz Fong-Jones — Honeycomb

A failure in their Software-Defined Networking (SDN) configuration system required manual recovery.

  Google

[…] when Kubernetes is involved, the number of alert sources can skyrocket quickly. This article will reflect on some common causes of alert fatigue and share tips to help reduce it.

  Nate Matherson — DZone

Meta has a special system to warn servers about power outages, giving them 45 seconds of battery power to finish things up and get ready to shut down.

  Raghunathan Modoor Jagannathan, Sulav Malla, and Parimala Kondety — Meta

This is an approachable explanation of the Paxos algorithm with examples, diagrams, and code.

  Martin Fowler

But what does reliability mean for people outside of engineering? And how does it translate into best practices for other teams?

  Emily Arnott — Blameless

“The Practice of Practice” is a concept from improvisational music. This article artfully applies the idea to the practice of incident response.

  Matt Davis — Blameless

I haven’t heard of this technique being used before, assigning alerts to on-call folks in round-robin order as they come in. I wonder if there’s a reason for that…

  Hannah Culver — PagerDuty

Raise your hand if you’ve been bitten by DNS before.

  Julia Evans

Ably processes a lot of messages, so when they have to redesign a core part of their architecture, it gets pretty interesting.

  Simon Woolf — Ably

If you ask any Site Reliability or DevOps engineer how they feel about a deployment plan with over 300 single points of failure, you’d see a lot of nauseous faces and an outbreak of nervous tics!

Nevertheless, that was the best design. Read on to find out why.

  Robert Barron

Slack had three separate incidents while trying to deploy DNSSEC for slack.com. This article goes into deep detail on what went wrong each time and what they learned.

Yes, it was an oversight that we did not test a domain with a wildcard record before attempting slack.com — learn from our mistakes!

  Rafael Elvira and Laura Nolan — Slack

The specializations outlined in this article include:

• The Educator
• The SLO Guard
• Infrastructure architect
• Incident response leader

  Emily Arnott — Blameless

If you had to design a WhatsApp today to support its current load, how would you go about it? Here’s one possible design.

  Ankit Sirmorya — High Scalability

Yesterday I asked on Twitter why you might want to run your own DNS servers, and I got a lot of great answers that I wanted to summarize here.

  Julia Evans

In this podcast interview, find out more about why Courtney Nash created the VOID and how posting an incident report can benefit your company. Transcript available.

  Mandy Walls (with guest Courtney Nash) — Page it to the Limit

Drawing on Cynefin, this article explains why debugging by feel and guesswork won’t suffice anymore; we need to be methodical.

  Pete Hodgson — Honeycomb

There are way too many gorgeous, mind-blowing ways for incidents to occur without a single change to code being deployed.

That last hot take is the kicker: even if you don’t do a code freeze in December (in the US), you’ll still see a lot of the same pitfalls as you would have if you did.

  Emily Ruppe — Jeli

Ah, IaC, the tool we use to machine-gun our feet in a highly-available manner at scale. This analysis of an incident from back in August tells what happened and what they learned.

  Stuart Davidson — Skyscanner

By establishing a set of core principles (Response, Observability, Availability and Delivery) aka our “ROAD to SRE”, we now have clarity on what areas we expect our SRE team should be focusing on and avoiding a common pitfall of becoming another platform or Ops team.

  Bruce Dominguez

In this blog post, we’ll look at:

• The advantages of an SRE team where each member is a specialist.
• Some SRE specialist roles and how they help.

  Emily Arnott — The New Stack

I love these “predictions for $YEAR” posts. What are your predictions?

  Emily Arnott — Blameless

Deployment Decision-Making during the holidays amid the COVID19 Pandemic

A sneak peek into my forthcoming MSc. thesis in Human Factors and Systems Safety, Lund University.

  Jessica DeVita (edited by Jennifer Davis) — SysAdvent

This article covers what to do as an incident commander, how to handle long-running incidents, and how to do a post-incident review.

  Joshua Timberman — SysAdvent

So in this post I’m going to go over what makes a good metric, why data aggregation on its own loses resolution and messy details that are often critical to improvements, and that good uses of metrics are visible by their ability to assist changes and adjustments.

  Fred Hebert

Here’s a great tutorial to get started with eBPF through a (somewhat convoluted) “Hello World” exercise.

  Ania Kapuścińska (edited by Shaun Mouton) — SysAdvent

The concept of engineering work being about resolving ambiguity really resonates with me.

  Lorin Hochstein

This appears to have caused a problem with Microsoft Exchange servers. Maybe this belongs in the Outages section…

  rachelbythebay