Need another team to do something fast? Just use this one weird trick: declare an incident! This article explains why the obvious solution (gating incident declaration) isn’t a good idea.

  Brent Chapman

Ethics are relative, right? This article is full of genuinely useful tips and framings.

  Thomas A. Limoncelli — ACM Queue

Whoa. It’s been quite a few years since my last run-in with an overfull conntrack table, and this is a fun new twist.

  Jorrick Sleijster — Adyen

For eight years I ran SRE for a storage system measured in exabytes. The dashboard I checked every morning shrank to seven numbers. Here they are.

  Sridhar Rajarao

Traditional observability monitors execution. LLM observability must monitor behavior.

  Barnadeep Bhowmik

This one has a lot of great detail on how their approaches to quota management failed and how they iterated.

  Dhyanam Vaidya, Prathamesh Deshpande, and Mike Ma — Uber

This article uses Voyager 1, whose engineers just shut down another instrument to conserve its steadily-decaying power, as an extended analogy for graceful degradation.

  Robert Barron

Celebrating heroes in incident response can incentivize further heroics. That can prevent the kind of growth that will improve incident response overall.

  Brent Chapman

What might happen when we quickly adopt LLMs and make sweeping changes in our complex systems?

  Fred Hebert

Most teams log, but log badly: wrong severity levels, no trace IDs, inconsistent fields, and logs siloed from traces.

  Ashwini Dave — DZone

If you had to explain to a neighbour why your organisation is so safe, and generally works well, what would you say?

It’s all about people. I really enjoyed the quote from Charles Billings on principles for automation.

  Steven Shorrock

Type conversion in aviation involves an experienced pilot training on a new kind of aircraft. This article draws a parallel to transitioning to a new job as an SRE.

  Bill Duncan

Some big names in this Q&A, and they share a couple of delicious morsels.

  Sam Salter — Uptime Labs, with John Allspaw and Beth Adele Long

A super-engaging deep-dive.

This investigation is a useful reminder: running boring technology in a non-standard way is a risk.

  Alex Chan — Tailscale

A handy guide on topology constraints in Kubernetes, with a worked example.

  Andre Newman — Gremlin

We may improve velocity by handing off tasks to LLM agents, but can that impact resilience?

  Courtney Nash — Resilience in Software Foundation

Fatigue and burn-out are reliability risks. Fatigue and burn-out are reliability risks. I champion this idea in my SRE practice constantly, and I hope you do too.

  Brent Chapman

A fun read on how to build control planes for large-scale systems, with some great tidbits on the inner workings of EC2 and Aurora DSQL.

  Zak van der Merw

A harrowing incident story underlining the importance of expertise and experience.

  Hamed Silatani — Uptime Labs

An SRE comes to terms with the way LLM agents are changing our field: what works well, what still requires human involvement, and what the future may look like.

  Bill Duncan

Recent outages at Tailscale, jsDelivr, ServiceNow, and IPinfo show the same failure pattern: certificate automation broke quietly, while the expiry date kept moving closer.

Bonus: they include links to several write-ups of related incidents.

  TokenTimer

Our solution treats infrastructure state as a traversable graph and lets a pathfinding algorithm discover recovery sequences at runtime.

Whoa, cool trick!

  Pragya Mehta and Sai Samant — Stripe

Their event-oriented system was based on Google Pub/Sub with its 99.95% SLA, but their own SLA was 99.99%. To resolve that, they moved toward an active-active architecture, load-balancing across 2 message brokers.

There’s an interactive simulation of their algorithm midway through that’s fun to play with!

  Patrick Hamann and Mike Fisher — incident.io

It’s not enough to define an incident process. You have to spin up and maintain an entire incident management program.

  Brent Chapman

Honeycomb pulls back the curtain a bit to delve into how LLM agents change the way their product is used, and how their query patterns differ from humans’. It’s especially interesting that increasing agent usage has not correlated with decreasing human usage.

  Austin Parker — Honeycomb

I like the approach here, especially measuring both the positive and negative outcomes.

Good SRE practice is about evidence, not enthusiasm.

   Neel Shah — DZone

The kernel’s route cache: a hidden reliability killer. This is a really intriguing case of self-sustaining impact.

  Ray Chen — Railway

…we rely on formal verification, and this is how consensus algorithms are built today. We define a model that we can mathematically prove to be correct, and then we… translate this perfect, platonic thing into code.

  TW Lim — Antithesis

I love that this starts with the user. Monitor what matters to your users, and alert on what you can action.

  Omar Ghader

First time I’ve heard of systemd’s PrivateTmp feature. Neat!

  Chris Siebenmann

I thought it would be a useful exercise to brainstorm some of the differences in focus between what I’ll call the traditional view of reliability, and the resilience engineering view.

It’s short (just a table), but it definitely made me think.

  Lorin Hochstein

Spotify has had some difficulty around podcast publishing, and they shared this analysis of the worst incident.

  Jim Whitehead, Ulrik Mikaelsson, John Lagomarsino, and Saunak Jai Chakrabarti — Spotify

…and here’s where it gets interesting. This post shares the user point of view on the Spotify issues, including fact-checking their published timeline.

  Gergely Orosz — The Pragmatic Engineer

New incident role unlocked: the incident tech lead. I enjoyed the description of the interplay between the tech lead and the incident commander.

  Brent Chapman

This one goes hard: if you try to reduce your incident count, your system will become less reliable, not more. Aim for more incidents, handled well.

  Tim Irving

There’s some brutal honesty in here that I find refreshing, especially around the impact on incidents and incident response.

  Liz Fong-Jones — Honeycomb

Here’s what I’ve learned about what actually happens inside a team when something breaks, how teams often reflect, and how we think about accountability without losing a blameless culture.

  Karan Nagarajowda — Uptime Labs

agent sprawl is now a production reliability crisis, and the SRE discipline does not yet have governance frameworks for it.

   Ajay Devineni — DZone

The premise: read replicas can help you scale read load, but they introduce complexity. The article goes into the problems they ran into and how they dealt with them.

  Johanna Larsson — incident.io