The section on fire department metrics does an incredible job of explaining why MTTR isn’t a useful metric.

  Brent Chapman

Full disclosure on this one, the author is my former (awesome) boss, and I think I may be that staff engineer they mentioned…

  Reid Savage — Honeycomb

Rootly grapples with how to evolve their PR review process with the increase in LLM-generated code.

The harder bugs hide in context, shared boundaries, and rollout paths.

  Quentin Rousseau — Rootly

The ladder does not end at SEV-3. It does not end at SEV-4. It ends somewhere below, in a category we have decided not to name.

  Tim Irving

The shift to distributed systems over monoliths can increase complexity. We don’t need to go back to the old way, says this article, but we do need to build with operability in mind.

  Vivek Kadam — Communications of the ACM

Every service catalog is declared by hand and drifts from reality within weeks. That was a productivity tax when humans read it. Now that agents act on it, a stale catalog is a production risk.

I’m not so sure about the solution offered, but the risk is real. Humans can already act on incorrect service catalog information, but agents can do it much more quickly.

  Spiros Economakis

What you’ll learn in this post isn’t a success story, it’s a learning journey. We’ll walk through the architecture decisions that enabled scale, the production challenges that tested those decisions, the optimization methodology that guided us through, and the lessons that apply to any distributed system.

  Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva, and Nathan Fisher

A summary of Uptime Labs’s Incident Fest, with a lot of interesting tidbits around AI, incidents, and the evolving interplay between them.

  Sam Salter — Uptime Labs

People hold post-incident reviews for three separate purposes. When the people that care about each one collide, things can go off the rails.

  Brent Chapman

In December, Honeycomb had a major incident, and they posted a pretty detailed write-up on their status page. That was just an interim report though, and this post goes into a ton more detail.

  Fred Hebert — Honeycomb

They had a weird problem, and they only really got to the bottom of it when they zoomed out and looked at the effects at the fleet level.

  Nathan Bronson — OpenAI

Is the human reviewer able to live up to the assurance they’re supposed to provide?

They are being asked to catch an error at the one moment they have the least context to catch it.

  Dan Leiva — CEOWORLD Magazine

The problem is, as an industry we more often than not mistake capturing and archiving information for developing meaningful insights.

  Will Gallego — Resilience in Software Foundation

AI lowers the barrier to entry. True. But it also lowers the barrier to overcommitment.

The question isn’t “Can AI help us build this faster?” The question is: “Should we own the infrastructure required to keep this alive for the next five years?”

   Bru Woodring — Prismatic

Cross-team latency problems are accountability problems, not just profiling problems. An SLO contract is one way to solve this.

   Ujjwal Gulecha — DZone

A guide for building an incident management process at a small company, with a focus on what not to include from the start.

  Tim Irving

Blame shuts down learning. That’s true whether the blame comes from a manager, from a peer, or from self-blame.

  Brent Chapman

This article argues that when we use AI to move faster, we’re limited by the “weak link” in our sociotechnical system: humans.

  Hamed Silatani — Uptime Labs

Two cool ideas in this one that stood out to me: identity diversity versus cognitive diversity and the practice of having everyone write up what they think happened in an incident before the retrospective meeting.

  Nick Travaglini — Honeycomb

Every GCP resource and API have quotas. In a big organization, you can start having production incidents due to hitting quotas you didn’t know about in projects you have never touched before.

  Aksel Allas — Coop Norge SA

Maybe that complex HA system is difficult enough to operate that a single-host solution would be more reliable.

  Jos Visser

LLMs enable interesting new use cases, like writing single-use code for use during incidents or testing.

  Oren Eini

A good introduction to formal verification and how LLMs can help. Formal verification may be more practical now that LLMs can help us write the proofs.

  Fernanda Graciolli and Nada Amin — ACM Queue

Oh, I’ve definitely felt that pull to debug as an IC. Gotta either hand over the IC reins or, as this article recommends, find a good tech lead.

  Brent Chapman

If your three data types can’t be joined programmatically today, an AI layer on top won’t fix that; it’ll just be confused faster.

  Pruthvi Raj Seknametla — HackerNoon

In this article, we’ve compiled a selection of tips we wish we had known the first time we picked up the pager or bore the BlackBerry.

  Uptime Labs

Me too. I do so much of my learning from an incident while I’m trying to write about it.

  Lorin Hochstein

The level of candor in this one is commendable. By all rights the maintenance itself went well — the incident was in the communication leading up to it.

  Fred Hebert — Honeycomb

This deep debugging story has a satisfying ending, and I can really feel the level of effort and detective work it took to get there.

   Deanna Lam, Diretnan Domnan, and Matt Lewis

How we made custom instrumentation blazing fast, simple, and data-centric

The answer was not just to throw AI at it.

  Jean-Mark Wright

We were curious whether AI could help us safely evolve a critical production system. This post is about what worked, what didn’t, and what we learned along the way.

I like their approach: AI is a tool only; powerful but not the whole solution.

  Arnold Wakim — Datadog

[…] the fix isn’t “train your engineers to write better status updates.” The fix is to stop asking your engineers to write them, and start asking the right people instead.

  Brent Chapman

A satisfying scaling story where every fix came from looking more closely at the system — Kafka head-of-line blocking, a clumpy scheduler, and an active-active API that silently doubled latency for half of all partitions.

  Dave Baxter — Cloudflare

Some good examples of risks in here, along with an interesting tendency to blame “user error”.

  Prakshal Doshi — HackerNoon

Satellites present unique reliability constraints like limited data uplink windows and the risk of bricking a very expensive piece of equipment.

Author:

This looks fun! It’s a free virtual event on July 8.

  Uptime Labs

This article does a really great job of building up an explanation of feedback-based control and the difference between edge-triggered and level-triggered systems.

  Fatih Arslan — PlanetScale

An open letter to software researchers to study incident response in software systems. It’s so cool how the author translates incident response concepts to researchers who may not be familiar, with examples.

  Lorin Hochstein

An important concept: a user’s perception of your average outage duration is weighted and won’t match a flat average MTTR.

  Marc Brooker