It’s not enough to define an incident process. You have to spin up and maintain an entire incident management program.

  Brent Chapman

Honeycomb pulls back the curtain a bit to delve into how LLM agents change the way their product is used, and how their query patterns differ from humans’. It’s especially interesting that increasing agent usage has not correlated with decreasing human usage.

  Austin Parker — Honeycomb

I like the approach here, especially measuring both the positive and negative outcomes.

Good SRE practice is about evidence, not enthusiasm.

   Neel Shah — DZone

The kernel’s route cache: a hidden reliability killer. This is a really intriguing case of self-sustaining impact.

  Ray Chen — Railway

…we rely on formal verification, and this is how consensus algorithms are built today. We define a model that we can mathematically prove to be correct, and then we… translate this perfect, platonic thing into code.

  TW Lim — Antithesis

I love that this starts with the user. Monitor what matters to your users, and alert on what you can action.

  Omar Ghader

First time I’ve heard of systemd’s PrivateTmp feature. Neat!

  Chris Siebenmann

I thought it would be a useful exercise to brainstorm some of the differences in focus between what I’ll call the traditional view of reliability, and the resilience engineering view.

It’s short (just a table), but it definitely made me think.

  Lorin Hochstein

Spotify has had some difficulty around podcast publishing, and they shared this analysis of the worst incident.

  Jim Whitehead, Ulrik Mikaelsson, John Lagomarsino, and Saunak Jai Chakrabarti — Spotify

…and here’s where it gets interesting. This post shares the user point of view on the Spotify issues, including fact-checking their published timeline.

  Gergely Orosz — The Pragmatic Engineer

New incident role unlocked: the incident tech lead. I enjoyed the description of the interplay between the tech lead and the incident commander.

  Brent Chapman

This one goes hard: if you try to reduce your incident count, your system will become less reliable, not more. Aim for more incidents, handled well.

  Tim Irving

There’s some brutal honesty in here that I find refreshing, especially around the impact on incidents and incident response.

  Liz Fong-Jones — Honeycomb

Here’s what I’ve learned about what actually happens inside a team when something breaks, how teams often reflect, and how we think about accountability without losing a blameless culture.

  Karan Nagarajowda — Uptime Labs

agent sprawl is now a production reliability crisis, and the SRE discipline does not yet have governance frameworks for it.

   Ajay Devineni — DZone

The premise: read replicas can help you scale read load, but they introduce complexity. The article goes into the problems they ran into and how they dealt with them.

  Johanna Larsson — incident.io

Amazing idea: negative time to detection. It’s when you know an incident is coming even before the impact actually begins. Some incident management and metrics systems aren’t designed to track it, and some incident processes ignore or even penalize it.

  Tim Irving

The key question is, when that new code breaks in production at 3am, how well can the on-call engineers debug it?

  Brent Chapman

This article focuses heavily on how Honeycomb built and tested a plan for what I can personally assure you must have been a very complex migration. I especially like how they drew lessons from their incident late last year.

  Josh Parsons — Honeycomb

An in-depth introduction to sharding in PostgreSQL. Sure, you probably know all about sharding, but I definitely learned some interesting bits from this one even so.

  Ben Dicken — PlanetScale

  This article is published by this issue’s sponsor, but their sponsorship did not influence its inclusion in the newsletter.

I started out ready to hate this one, but by the end, I came around; there’s a lot to think about. It’s about how coding agents change the economics of saying no or yes to taking on projects.

Cheap to write is not the same as cheap to own

  Dalia Abuadas — GitHub

In this commercial airliner near miss, a system designed for reliability (alpha floor protection) was the cause of an incident (uncommanded pitch down), giving an excellent case study in automation.

Do we know when it works, how it works, how to get the most out of it, and how to find a way around it if it turns against us?

  Mentour Pilot

Conventional wisdom says a database makes a bad queue and you should reach for a real broker but these folks went the other direction, tearing out RabbitMQ in favor of polling MongoDB. They included a section at the end of everything they had to build themselves to make polling work.

  Pavel Zavialov — DevOps.com

A detailed description of their approach, including how they tested it and the weak points they’re still iterating on.

  Akhilesh Rao Meesala — HackerNoon

The section on fire department metrics does an incredible job of explaining why MTTR isn’t a useful metric.

  Brent Chapman

Full disclosure on this one, the author is my former (awesome) boss, and I think I may be that staff engineer they mentioned…

  Reid Savage — Honeycomb

Rootly grapples with how to evolve their PR review process with the increase in LLM-generated code.

The harder bugs hide in context, shared boundaries, and rollout paths.

  Quentin Rousseau — Rootly

The ladder does not end at SEV-3. It does not end at SEV-4. It ends somewhere below, in a category we have decided not to name.

  Tim Irving

The shift to distributed systems over monoliths can increase complexity. We don’t need to go back to the old way, says this article, but we do need to build with operability in mind.

  Vivek Kadam — Communications of the ACM

Every service catalog is declared by hand and drifts from reality within weeks. That was a productivity tax when humans read it. Now that agents act on it, a stale catalog is a production risk.

I’m not so sure about the solution offered, but the risk is real. Humans can already act on incorrect service catalog information, but agents can do it much more quickly.

  Spiros Economakis

What you’ll learn in this post isn’t a success story, it’s a learning journey. We’ll walk through the architecture decisions that enabled scale, the production challenges that tested those decisions, the optimization methodology that guided us through, and the lessons that apply to any distributed system.

  Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva, and Nathan Fisher

A summary of Uptime Labs’s Incident Fest, with a lot of interesting tidbits around AI, incidents, and the evolving interplay between them.

  Sam Salter — Uptime Labs

People hold post-incident reviews for three separate purposes. When the people that care about each one collide, things can go off the rails.

  Brent Chapman

In December, Honeycomb had a major incident, and they posted a pretty detailed write-up on their status page. That was just an interim report though, and this post goes into a ton more detail.

  Fred Hebert — Honeycomb

They had a weird problem, and they only really got to the bottom of it when they zoomed out and looked at the effects at the fleet level.

  Nathan Bronson — OpenAI

Is the human reviewer able to live up to the assurance they’re supposed to provide?

They are being asked to catch an error at the one moment they have the least context to catch it.

  Dan Leiva — CEOWORLD Magazine

The problem is, as an industry we more often than not mistake capturing and archiving information for developing meaningful insights.

  Will Gallego — Resilience in Software Foundation

AI lowers the barrier to entry. True. But it also lowers the barrier to overcommitment.

The question isn’t “Can AI help us build this faster?” The question is: “Should we own the infrastructure required to keep this alive for the next five years?”

   Bru Woodring — Prismatic

Cross-team latency problems are accountability problems, not just profiling problems. An SLO contract is one way to solve this.

   Ujjwal Gulecha — DZone

A guide for building an incident management process at a small company, with a focus on what not to include from the start.

  Tim Irving