the idea that the highest ROI for risk reduction work is in the highest severity incidents is not a fact, it’s a hypothesis that simply isn’t supported by data.

  Lorin Hochstein

Incidents are bad, so should we try to have fewer of them? This article challenges the assumptions contained within that goal and suggests other ways to frame one’s thinking.

  Hamed Silatani — Uptime Labs.

This guide goes deeply into the details of how Prometheus uses memory, and then it shows you how to get a handle on it.

  Vladimir Guryanov — Palark

This article discusses the DNS-related challenges encountered at Mercari on our Kubernetes clusters and the significant improvements achieved by implementing Node-Local DNS Cache.

  Satyadarshi Sanu — Mercari

In this post we’ll explore the fundamentals of distributed consensus, compare the dominant consensus algorithms Paxos and Raft, and examine recent implementations like Kafka Raft.

  Narendra Reddy Sanikommu — DEV

A discussion of two techniques the folks at Cash App used to improve their reliability: adopting a two-cluster topology with Kubernetes, and using Amazon’s Fault Injection Service to simulate the failure of an availability zone.

  Dustin Ellis, Deepak Garg, Ben Apprederisse, Jan Zantinge, and Rachel Sheikh — Amazon

Reading this one taught me a couple of techniques I wasn’t aware of for finding queries in need of optimization in MySQL.

  Vinicius Grippa — Readyset

Ouch — and a great learning opportunity for all of us:

When our backend circuit breakers triggered, aggressive websocket reconnect logic initiated on every connected client at once, further overwhelming an already stressed database.

  Jake Cooper — Railway

Rollbacks don’t always return you to a previous system state. They can return you to a state you’ve never tested or operated before.

  Steve Fenton — Octopus Deploy

This article explains the math of burn rate alerting and gives well thought out reasoning or why burn rates are better.

  James Frullo — Datadog

This hot take is worth thinking about: what do you want to get out of assigning incident severity levels, and is it working?

  Hamed Silatani — Uptime Labs

Less defense, and more about how to best cope with a code freeze and avoid the downsides when you’ve got no choice.

  Tom Elliott

MTTI in this case is Mean Time to Isolate. How long are you taking to figure out what system component is at the heart of an incident? What does MTTI say about your system, and what can you do about it?

  Old School Burke

This article doesn’t answer the question in its title concretely, but it does give one a lot to think about. It also shares some ideas for how to cope with the potential challenges identified.

  Sylvain Kalache — LeadDev

This one starts off as a review of a workbook on root cause analysis by the UK Health and Safety Executive. Then it raises concerns about RCA-based reasoning and contrasts with a different model based on resilience engineering.

  Lorin Hochstein

I wrote this article in response to Azure’s post, Introducing Azure SRE Agent. There’s a lot we can learn from the example agent interactions that Microsoft chose to share.

  Lex Neva

Datadog has fully merged their SRE and Security teams.

In this post, we’ll look at essential elements of SRE and security, the benefits we’ve realized by combining the two disciplines, and what that approach looks like for us.

  Bianca Lankford — Datadog

I love the way this article describes three different audiences for your communication during incidents. It describes what each audience is looking for and gives both positive and negative examples of how to communicate with them.

  Hamed Silatani — Uptime Labs

My favorite part of this article is the section on where to run your load tests: production, staging, or something else?

  Tom Elliot

What is complexity? This article gives a clear definition and breaks down the qualities one can find in a complex system. Then it goes over various methods of dealing with that complexity.

  Teiva Harsanyi — The Coder Cafe

Cloudflare has a history of doing some pretty interesting things with sockets in Linux — and taking us along for the journey with highly-detailed explanations. This article is no exception, sharing the unique challenges encountered when restarting processes that handle UDP streams.

  Marek Majkowski

This article examines the standard friday deploy prohibition and ultimately pushes back.

Ok… but why not?

  Adrien Guéret — OpenClassrooms

This article introduces the STAMP (System-Theoretic Accident Model and Processes) framework being adopted at Google, after first explaining the shortcomings in traditional SRE practices that prompted Google to adopt STAMP.

  Jorge Lainfiesta — Rootly

I really love this framing of what’s wrong with picking a single root cause.

  Lorin Hochstein

Why don’t we look for the root cause of a successful outcome?

  Hamed Silatani — Uptime Labs

They took a great deal of care to avoid the potential pitfalls of using an LLM in this way, and they share a lot of detail about the steps they took.

  Tran Le, Till Pieper, and Gillian McGarvey — Datadog

After dealing with a late-night outage with surprisingly small impact, I got thinking about how you would know if you were working too hard to guarantee uptime.

  Tom Elliott

In this article, learn how the 4 R’s — robust architecture, resumability, recoverability, and redundancy — enhance reliability in AI and ML data pipelines.

   Sidhant bendre — DZone

In this article, I’ll delve into the challenges we encountered and the strategies we employed to manage operator upgrades for stateful workloads like Elasticsearch. Additionally, I’ll detail how we modified the ECK [Elastic Cloud on Kubernetes] operator to facilitate a more resilient side-by-side upgrade process.

  Abhishek Munagekar — Mercari

In this piece, I’ll delve into four macro challenges facing observability today, explore strategies that are emerging across the industry to address them, and offer my perspective on the trajectory of this crucial domain in the year to come.

  Andrew Mallaband

A deep-dive into a pretty nifty system for enumerating and provisioning a rack of servers, complete with PXE-based Debian headless installation using an auto-generated preseed file. It also uses Claude to figure out what state a server is in from a screenshot obtained from the BMC.

  Charith Amarasinghe — Railway

Koreo is a new open source tool for orchestrating Kubernetes infrastructure at a higher level than standard tools like Helm.

Koreo is a fairly complex tool, so it can be difficult to quickly grasp just what exactly it is, what problems it’s designed to solve, and how it compares to other, similar tools. In this post, I want to dive into these topics and also discuss the original motivation behind Koreo.

  Tyler Treat

This one is about understanding how work actually happens in our sociotechnical systems (versus how we imagine it). This has implications for how we learn from incidents and how we design corrective actions.

  Lorin Hochstein

The myth is:

The underlying and often unexamined assumption for the benefits of automation is the notion that computers/machines are better at some tasks, and humans are better at a different, non-overlapping set of tasks.

This article lays out several pitfalls to this approach, with references.

  Courtney Nash

Wow, I seriously love this one. It’s written in an a very approachable style that’s easy to understand from the outside. It lays a series of cringe-worthy contributing factors that could happen to any of us, making them a great learning opportunity.

  Spotify

This is the first time I’ve come across the term “grounding” in incident response, and I like it!

At the core of our vision lies the principle of grounding, drawn from safety-critical systems like aviation and the fire service industries. Grounding is the process of maintaining a shared understanding among team members throughout the course of an incident.

  Uptime Labs

I really like the idea of using formal modeling on distributed systems. Datadog explains how they did it when building a new message queuing service.

  Arun Parthiban, Sesh Nalla, and Cecilia Wat-Kim

I found this to be a really useful primer on the new EU AI regulation. It does transition into a sales pitch toward the end, but the pre-pitch content is substantial.

  Chris Evans — incident.io

A classic example of Lorin’s Law: work intended to improve reliability was at the heart of this incident.

  Railway

Feature flags are incredibly useful, but they have some gotchas too.

  Tom Elliott

More potential problems to watch out for with feature flags, but this one ends by emphasizing that feature flags are still an important tool. Bonus points for a Knight Capital Incident mention.

  Ian Vanagas