I’ve seen this so many times. If declaring an incident gets in the way of responding to it, something’s wrong.

  Hugo van der Horst

They have a maximum concurrency in Lambda, and they want to ensure that no one misbehaving client can exhaust it on its own.

  Daniil Romashov

This one’s really interesting. The PLC refused to enable generators because the standby utility power was live. However, that standby utility power couldn’t be used because the breaker was purposefully opened during maintenance. I’m sure thorough failure testing was completed, it’s just that this kind of scenario hadn’t occurred to them.

  Google

The fire service doesn’t track mean time to put out a fire.  Why should we measure our MTTR?

  Brent Chapman

Wow. Between this, the difficulty in reverting back to standard coreutils, and ubuntu’s schedule in publishing a fix for CVE-2026-98432, I’m having some second thoughts on Ubuntu. And no, Chris, I don’t think you should be expected to file a bug report the sudo-rs stuff.

  Chris Siebenmann

It’s like the security equivalent of Lorin’s law: a new process put in place with the goal of improving security (for a subset of users) itself was a the core of a security incident.

I’m really looking forward to uMudhra’s incident report, due tomorrow.

  Lorin Hochstein

An overview of queues and what they’re good for with a reliability engineering lens.

  Alex Ewerlöf

Runbook as hand-off mechanism. Runbook as a pointer to who knows more. Runbook as an embodiment of hard-earned lessons from the past. This one has me thinking about runbooks in new ways.

  Tim Irving

Archaeological layers of P0 incident action items? There’s a better way.

  Brent Chapman

Wow, the original for this article was one I included all the way back in issue #16. I think I’ve quoted the concepts from that one a million times.

  J. Paul Reed

At incident.io, agents now go beyond coding their mobile app changes, they actually run the app in a simulator and provide visual evidence of the change for a human to review.

  Grayson Ho — incident.io

Getting Debezium to stream the first PostgreSQL change into Kafka is straightforward; keeping it correct through failover is the interesting part.

  Frane Jelavic — ShiftMag

This one shows the drawbacks of focusing on a root cause through a really cool lens: a Five Whys analysis of the sinking of the Titanic.

  Steve Mynett — Rootly

Giving an agent more context can make it less effective. Here’s Meta’s technique to mitigate that.

  Plawan Rath, Akhil Jindal and Ashwini Verma — Meta

Migrating the busiest 0.00001% of repositories to Azure took some careful work and consideration of bottlenecks.

  Eleanor White and Victoria Dye — GitHub

My favorite section is the one about achieving higher reliability by coordinating between their client library and the server side.

  Matt Hammond — Ably

Events like the recent compromise of CA signing keys are discussed out in the open, on Mozilla’s Bugzilla. Right now, you can watch the post-incident analysis as it unfolds.

  Mozilla

This article gives a really easy-to-follow explanation of the consistency issues you get when writing across two systems and how to solve them with an outbox pattern.

   Rahul Tewari — DZone

An explanation of why a service catalog is so important, especially during incidents.

Twenty minutes in, you haven’t started diagnosing anything. You’ve been doing archaeology.

Bonus: part two on keeping your service catalog up-to-date.

  Bill Duncan

The Firebase SDK caused app-side crashes due to a server-side data change. This article focuses on Google’s response, and I also recommend reading the writeup on the Firebase blog that Gergely links to.

  Gergely Orosz — The Pragmatic Engineer

LLM agents behave fundamentally differently from deterministic programs, and we can’t just expect them to work like a natural language programming language.

  Uwe Friedrichsen

This one delves into the design choices involved in running tons of little databases at the edge efficiently.

  David Laban and Lambros Petrou — Cloudflare

A seasoned emergency responder and computer incident management engineer shares a rare perspective on the similarities and differences between the two fields.

  Brent Chapman

Now I start a chat session with an AI model, give him a personality, most often it is an old and grumpy but well-mannered distinguished engineer…

  Hrvoje Rancic — ShiftMag

Today, I learned that “hot take” does not mean what I thought it meant. Specifically, it does not mean “telling it like it is and not holding back”. My deepest apologies for last week’s issue, Alex.

  Alex Ewerlöf

Grey failure, feedback loops, retry storms, and a mis-aligned safeguard, complexity, plus more — there’s a lot to dig into in this incident.

  Lorin Hochstein

They tested using Jev for SRE tasks paired only with a deterministic scaffolding, not an LLM.

  Yiming Su, Saad Mohammad Rafid Pial, Jackson Clark, and Tianyin Xu — SREGym

A case that AI doesn’t quietly swap out human work one-for-one, it reshuffles our roles into something harder. I see a strong thread of the Ironies of Automation throughout this one.

  John Allspaw — Adaptive Capacity Labs

A short interview with a seasoned incident commander. My favorite part was question 3.

  Simona Ursan — Uptime Labs

We’re going to do this via a metric that nobody’s ever heard of before, called “Mean Time to Stop Sucking” (MTSS). Just like in last week’s post, I’m going to argue that, despite the name, this is a real metric that you ought to be tracking in your autoscaler, and then explain why nobody does.

  drmorr — ACRL

How we capture real production database traffic at Airbnb and replay it offline to load-test, plan capacity, and de-risk upgrades.

  Zuofei Wang

Alert on what matters to your customers, not what you have easy metrics for. Reading this reminds me that I should set up a weekly review of on-call alerts to see which ones were actionable.

  Brent Chapman

This is rare: an article from an AI SRE solution vendor that speaks to the risk we take in adopting this kind of tool. They share quite a bit of detail on what they are doing to try to understand and improve their tool’s accuracy.

  Martha Lambert — incident.io

This was a fascinating incident! I love how Lorin shares his process for trying to understand Telstra’s incident report.

  Lorin Hochstein

No summary from me, I don’t want to ruin it. Well worth a read.

  Lorin Hochstein

Alex tells it like it is, and I loved it.

AI can explain it to you but it cannot understand it for you. That understanding is a key aspect of ownership.

The original version of this issue incorrectly used the term “hot take”.

  Alex Ewerlöf

Error budgets are about understanding and managing risk, and the risk equation is changing with LLMs.

if a material change depends on concentrated knowledge, single points of failure, or solely on AI models, that change is riskier than a similar change the team can explain and operate without them.

  Paul LaPosta — LeadDev

How do you measure kubernetes cluster utilization?

But I once had an economics professor who would respond whenever you mentioned a rate like productivity, efficiency, or utilization with a simple question: “Of what?” This was a polite way of saying: “Tell me the damn numerator and denominator, and do it right now.”

  Ian O’Gorman — ACRL

A code change was deployed before the schema change it depended on.

  Angelo Saraceno — Railway

Key business logic lived in the database as Groovy scripts, and analysts changed it live constantly. Why set up version control and CI/CD when everything worked fine? How they made the case is an interesting story.

  Dmitry Moskalyuk