This year’s VOID Report is out, and it’s well worth a read. The subtitle is “Exploring the Unintended Consequences of Automation in Software” which is a really good way to get me to read something!

  Courtney Nash — The VOID

A terraform change deleted a critical resource, and reviewers missed it because the plan was so big. Now they use Atlantis and Open Policy Agent to avoid accidental deletions of critical resources.

  Lin Du — InfoQ

When analyzing an incident, what can we learn when we assume that everyone did everything as well as possible?

  Lorin Hochstein

onsite technicians performing this planned network maintenance inadvertently unplugged several fibers that were adjacent to those in the work order, but still in use for production traffic

  Google

There’s a huge difference between four and five nines. There’s an especially interesting quote in this article that Google doesn’t think five nines is attainable in a commercial service.

  Diana Bocco — UptimeRobot

Here’s an interview with three SREs about what it’s like to be an SRE at IBM.

  IBM

I’ve been hearing about Observability 2.0 but didn’t know what it was all about. This article explains what it is and how it can help with cost.

  Charity Majors — Honeycomb
  Full disclosure: Honeycomb is my employer.

A cute little video pep talk for SREs. The site is actually real, too!

  Krazam

Like a mini Y2K, leap day came around again and left some technical glitches in its wake, as chronicled in this article.

  Gergely Orosz — The Pragmatic Engineer

This article discusses building failure management directly into our systems, using Erlang as a case study.

  Jamie Allen

Building on their experience with their previous load shedding library, Uber built a new one that requires no configuration.

  Jakob Holdgaard Thomsen, Vladimir Gavrilenko, Jesper Lindstrom Nielsen, and Timothy Smyth — Uber

These folks found a way to get tag names and values from other people’s AWS resources. I know this is more security- than SRE-related but the technique is just so cool!

  Daniel Grzelak — Plerion

How much does it cost to improve resilience? What’s the ROI? It’s fuzzy, but we still need to do it.

  Will Gallego

Check it out, it’s an entire SRE conference I was totally unaware of!

  SREday

It’s an SLI/SLO/SLA explainer, but with a twist: a pros and cons list for each of the three.

  Laura Clayton — UptimeRobot

A great reddit thread for some schadenfreude… and perhaps you’d like to share your own story?

  u/New_Detective_1363 and others — reddit

What an interesting cause for an incident: the service your customers have pointed your product at decides to block your requests, effectively DoSing your systems.

  Tomas Koprusak — UptimeRobot

The CAP theorem is useful as a theory, but what does it actually mean in practice?

  neda — ReadySet

Can a single dashboard to cover your entire system really exist?

  Jamie Allen

This one makes the case for having a group of specially-trained incident commanders to handle SEV-1 (worst-case) outages, separate from your normal ICs.

  Jonathan Word

This article lays out a strategy for gaining buy-in by making three specific, sequential arguments.

  Emily Arnott — Blameless

This article explores the varying ways that SRE is implemented through a set of 4 archetypes.

  Alex Ewerlöf

It turns out that assigning ephemeral ports to connections in Linux is way more complicated than it might seem at first glance, and there’s room for optimization, as this article explains.

  Frederick Lawler — Cloudflare

While deploying Precision Time Protocol (PTP) at Meta, we’ve developed a simplified version of the protocol (Simple Precision Time Protocol – SPTP), that can offer the same level of clock synchronization as unicast PTPv2 more reliably and with fewer resources.

  Oleg Obleukhov and Ahmad Byagowi — Meta

Far more than just a list of links, this article gives an overview of each topic before pointing you in the right direction for more information.

  Fred Hebert

Building on the groundwork laid out in our first article about the initial steps in Incident Management (IM) at Dyninno Group, this second installment will explore the practicalities of streamlining and implementing these strategies.

  Vladimirs Romanovskis

Software engineers and SREs should share a single on-call rotation as part of a single team, as this is where empathy for each other is built.

  Jamie Allen

I was pretty fuzzy on what HTTP/3 was all about, but this article set me straight.

  Roopa Kushtagi

An overview of the modulith pattern including reasons to choose modulith over microservices.

  Pier-Jean Malandrino

This article explores feedback loops formed out of various ways of responding to incidents that in turn increase the likelihood of more incidents. It took me a couple tries to get into this one, but it was well worth my effort.

  Steven Shorrock

Here, we’re going to outline some practical things you should consider when visiting on-call compensation and the incentives you create around it. We’ll also share how we approach this conversation here at incident.io.

  incident.io

This link-aggregation repo isn’t just about interviewing for SRE roles. It also links to resources on a ton of topics relevant to those starting out in SRE.

  @mxssl on GitHub

Cool trick: this paper uses counterfactual “should have” statements for good as a way of surfacing what incident investigators wish auditing was looking for. Click through for Fred Hebert’s synopsis of the paper.

  Fred Hebert (summary)   Ben Hutchinson, Sidney Dekker, and Andrew Rae (original authors) — Process Safety Progress

This article (part one in a series) follows the author’s journey to learn and improve incident management at their company.

  Vladimirs Romanovskis — Dyninno

In this blog post, we describe the journey DoorDash took using a service mesh to realize data transfer cost savings without sacrificing service quality.

  Hochuen Wong and Levon Stepanian — DoorDash

When just a few “regulars” are called in to handle every incident, you’ve got a knowledge gap to fill in your organization.

  David Ridge — PagerDuty

Dropbox expands into new datacenters often, so they have a streamlined and detailed process for choosing datacenter vendors.

  Edward del Rio — Dropbox

This is either nine things that could derail your SRE program, or a list of things to do with “not” in front of them — either way, it’s a good list.

  Shyam Venkat

We need enough alerting in our systems that we can detect lurking anomalies, but not so much that we get alert fatigue.

  Dennis Henry

A post about the importance of product in SRE, and how to make product and SRE first-class citizens in your Software Development Lifecycle.

  Jamie Allen

A relatively minor incident took a turn for the worse after the pilots attempted a close fly-by in an attempt to resolve it. I swear I’ve been in this kind of incident before, where I took risks significantly out of proportion to the problem I was trying to solve.

  Kyra Dempsey (Admiral Cloudberg)