The better these tools become at resolving routine incidents, the less practice human responders will get. And when an ambiguous, high-severity incident comes in that automation cannot solve, responding engineers will be in trouble.

I love the concept of comprehension debt described in this article.

  Sylvain Kalache

Another excellent write-up by my favorite writer of incident write-up write-ups. I especially like that last section.

Lorin also wrote more the next day.

  Lorin Hochstein

A VP’s question […] lands with the weight of the org chart behind it, and everyone in the channel feels it.

I especially liked the sections toward the end on how to mitigate a leader’s impact.

  Brent Chapman

Lowe’s has an intensive (and intense) SRE practice that provides client teams with a framework of tools to ensure reliability.

  Raghuprasanth Ravichandran, Joe Praveena A, and Pavan Palagiri

How far will the impact of an incorrect automated decision spread? Will it get integrated into the system and influence later decisions?

  Sai Sandeep Koneti — HackerNoon

A great primer on how pod disruption budgets work and why they’re critical for reliability.

  Andre Newman — Gremlin

They took a systematic approach using test-driven development. I enjoyed not only learning what went well, but where they ran into trouble.

  Edvinas Janusevicius — Checkly

Great advice if you’re training early-career engineers on incident response. This advice reminds me of some of the techniques that folks used with me when I was starting out.

  Karan Nagarajagowda — Uptime Labs

What can you do to shorten the time to detect an incident? Some great ideas in here, especially monitoring your company’s main web page for a sudden uptick in traffic.

  Brent Chapman

What an interesting incident! I recommend reading Azure’s write-up before reading Lorin’s excellent analysis.

  Lorin Hochstein

Distributed databases rarely fail in the clean, isolated ways described by component diagrams. They fail through timing gaps, stale metadata, ambiguous ownership, retry storms, incompatible health decisions, and overlapping maintenance activity.

   Varsha Ganesh — DZone

I love this concept of a “political incident”:

The subject was political incidents, by which I mean the ones where the severity arrives before the impact assessment does.

And ouch, I felt this bit:

You have spent forty minutes of the incident on the severity field.

  Tim Irving

Where can you safely use LLM agents, versus when you should keep things in human hands? This one has some good criteria to consider.

  Sai Joshitha Kathari — HackerNoon

I learned a lot about Git while reading this one. Speeding up Git clones in CI may not seem important, but it will when you’re trying to roll out a fix during an incident.

  Mike Thompson and Daniel Esponda — Datadog

Switching from their custom-written autoscaler to the new off-the-shelf option made sense, but it wasn’t a simple drop-in replacement.

  Samuel Yeboah, Francesco Di Chiara and Mingliang Liu — Netflix

Can we replace human code review with LLM-based reviews? This article lays out what an LLM can’t replicate, and I’d argue that these are the pieces that matter most for reliability.

  John Allspaw — Adaptive Capacity Labs

Need another team to do something fast? Just use this one weird trick: declare an incident! This article explains why the obvious solution (gating incident declaration) isn’t a good idea.

  Brent Chapman

Ethics are relative, right? This article is full of genuinely useful tips and framings.

  Thomas A. Limoncelli — ACM Queue

Whoa. It’s been quite a few years since my last run-in with an overfull conntrack table, and this is a fun new twist.

  Jorrick Sleijster — Adyen

For eight years I ran SRE for a storage system measured in exabytes. The dashboard I checked every morning shrank to seven numbers. Here they are.

  Sridhar Rajarao

Traditional observability monitors execution. LLM observability must monitor behavior.

  Barnadeep Bhowmik

This one has a lot of great detail on how their approaches to quota management failed and how they iterated.

  Dhyanam Vaidya, Prathamesh Deshpande, and Mike Ma — Uber

This article uses Voyager 1, whose engineers just shut down another instrument to conserve its steadily-decaying power, as an extended analogy for graceful degradation.

  Robert Barron

Celebrating heroes in incident response can incentivize further heroics. That can prevent the kind of growth that will improve incident response overall.

  Brent Chapman

What might happen when we quickly adopt LLMs and make sweeping changes in our complex systems?

  Fred Hebert

Most teams log, but log badly: wrong severity levels, no trace IDs, inconsistent fields, and logs siloed from traces.

  Ashwini Dave — DZone

If you had to explain to a neighbour why your organisation is so safe, and generally works well, what would you say?

It’s all about people. I really enjoyed the quote from Charles Billings on principles for automation.

  Steven Shorrock

Type conversion in aviation involves an experienced pilot training on a new kind of aircraft. This article draws a parallel to transitioning to a new job as an SRE.

  Bill Duncan

Some big names in this Q&A, and they share a couple of delicious morsels.

  Sam Salter — Uptime Labs, with John Allspaw and Beth Adele Long

A super-engaging deep-dive.

This investigation is a useful reminder: running boring technology in a non-standard way is a risk.

  Alex Chan — Tailscale

A handy guide on topology constraints in Kubernetes, with a worked example.

  Andre Newman — Gremlin

We may improve velocity by handing off tasks to LLM agents, but can that impact resilience?

  Courtney Nash — Resilience in Software Foundation

Fatigue and burn-out are reliability risks. Fatigue and burn-out are reliability risks. I champion this idea in my SRE practice constantly, and I hope you do too.

  Brent Chapman

A fun read on how to build control planes for large-scale systems, with some great tidbits on the inner workings of EC2 and Aurora DSQL.

  Zak van der Merw

A harrowing incident story underlining the importance of expertise and experience.

  Hamed Silatani — Uptime Labs

An SRE comes to terms with the way LLM agents are changing our field: what works well, what still requires human involvement, and what the future may look like.

  Bill Duncan

Recent outages at Tailscale, jsDelivr, ServiceNow, and IPinfo show the same failure pattern: certificate automation broke quietly, while the expiry date kept moving closer.

Bonus: they include links to several write-ups of related incidents.

  TokenTimer

Our solution treats infrastructure state as a traversable graph and lets a pathfinding algorithm discover recovery sequences at runtime.

Whoa, cool trick!

  Pragya Mehta and Sai Samant — Stripe

Their event-oriented system was based on Google Pub/Sub with its 99.95% SLA, but their own SLA was 99.99%. To resolve that, they moved toward an active-active architecture, load-balancing across 2 message brokers.

There’s an interactive simulation of their algorithm midway through that’s fun to play with!

  Patrick Hamann and Mike Fisher — incident.io