Blame shuts down learning. That’s true whether the blame comes from a manager, from a peer, or from self-blame.

  Brent Chapman

This article argues that when we use AI to move faster, we’re limited by the “weak link” in our sociotechnical system: humans.

  Hamed Silatani — Uptime Labs

Two cool ideas in this one that stood out to me: identity diversity versus cognitive diversity and the practice of having everyone write up what they think happened in an incident before the retrospective meeting.

  Nick Travaglini — Honeycomb

Every GCP resource and API have quotas. In a big organization, you can start having production incidents due to hitting quotas you didn’t know about in projects you have never touched before.

  Aksel Allas — Coop Norge SA

Maybe that complex HA system is difficult enough to operate that a single-host solution would be more reliable.

  Jos Visser

LLMs enable interesting new use cases, like writing single-use code for use during incidents or testing.

  Oren Eini

A good introduction to formal verification and how LLMs can help. Formal verification may be more practical now that LLMs can help us write the proofs.

  Fernanda Graciolli and Nada Amin — ACM Queue

Oh, I’ve definitely felt that pull to debug as an IC. Gotta either hand over the IC reins or, as this article recommends, find a good tech lead.

  Brent Chapman

If your three data types can’t be joined programmatically today, an AI layer on top won’t fix that; it’ll just be confused faster.

  Pruthvi Raj Seknametla — HackerNoon

In this article, we’ve compiled a selection of tips we wish we had known the first time we picked up the pager or bore the BlackBerry.

  Uptime Labs

Me too. I do so much of my learning from an incident while I’m trying to write about it.

  Lorin Hochstein

The level of candor in this one is commendable. By all rights the maintenance itself went well — the incident was in the communication leading up to it.

  Fred Hebert — Honeycomb

This deep debugging story has a satisfying ending, and I can really feel the level of effort and detective work it took to get there.

   Deanna Lam, Diretnan Domnan, and Matt Lewis

How we made custom instrumentation blazing fast, simple, and data-centric

The answer was not just to throw AI at it.

  Jean-Mark Wright

We were curious whether AI could help us safely evolve a critical production system. This post is about what worked, what didn’t, and what we learned along the way.

I like their approach: AI is a tool only; powerful but not the whole solution.

  Arnold Wakim — Datadog

[…] the fix isn’t “train your engineers to write better status updates.” The fix is to stop asking your engineers to write them, and start asking the right people instead.

  Brent Chapman

A satisfying scaling story where every fix came from looking more closely at the system — Kafka head-of-line blocking, a clumpy scheduler, and an active-active API that silently doubled latency for half of all partitions.

  Dave Baxter — Cloudflare

Some good examples of risks in here, along with an interesting tendency to blame “user error”.

  Prakshal Doshi — HackerNoon

Satellites present unique reliability constraints like limited data uplink windows and the risk of bricking a very expensive piece of equipment.

Author:

This looks fun! It’s a free virtual event on July 8.

  Uptime Labs

This article does a really great job of building up an explanation of feedback-based control and the difference between edge-triggered and level-triggered systems.

  Fatih Arslan — PlanetScale

An open letter to software researchers to study incident response in software systems. It’s so cool how the author translates incident response concepts to researchers who may not be familiar, with examples.

  Lorin Hochstein

An important concept: a user’s perception of your average outage duration is weighted and won’t match a flat average MTTR.

  Marc Brooker

Spontaneous swarming of responders might seem like a nuisance that breaks our tidy mental models of incident response, but it’s actually very powerful. It’s something to facilitate and encourage, not simply tolerate.

  Brent Chapman

The misconception is that the local assurances automatically combine to form a single end-to-end promise that spans brokers, processors, databases, outboxes, caches, webhooks, and external APIs.

   Irullappan irulandi — DZone

When a firmware issue caused reboots for firmware upgrades to take four hours(!), they had to find a solution.

  Giovanni Pereira Zantedeschi, Nnamdi Ajah, and Omar Sheik-Omar — Cloudflare

This one strikes a balance on AI that really speaks to me.

If you’re the one left holding the bag, you should generally get final say over what goes in that bag.

  Charity Majors

How Airbnb built a Kubernetes sidecar to deliver dynamic configuration reliably at scale.

  Bo Teng — Airbnb

In this post, we’ll walk through how we redesigned our Kubernetes-based PostgreSQL clusters for failover safety, how we balanced durability against latency, and what we learned while validating this approach through benchmarking and failure testing.

  Shree Sampath — Datadog

The failure mode on this one is really interesting, and the bit about “infinite blast radius” caught my eye.

  Sarat Mahavratayajula ,Vijay Sagar Gullapalli — VentureBeat

I’m enjoying this series so far, and I’m looking forward to reading the rest. It’s worth starting at part 1, but part 2 can stand on its own in a pinch.

  Uwe Friedrichsen

We build our systems against the usage patterns of human users, but agents fundamentally change the game.

   Vineet Bhatkoti — DZone

This is an interesting lens for exploring the risks that agents can introduce.

  Sayali Patil — VentureBeat

Great discussion in the comments! There’s a lot of variance in how much time people recommend. I personally tend to lean earlier — on-call is a great way to learn, and I can always reach out if I get stuck.

  u/modern_medicine_isnt and commenters — Reddit r/sre

A great into to the concept of metastable failures — and I recommend reading the original paper as well.

  Teiva Harsanyi

The real issue is that your company has made declaring an incident costly and risky for the person who does it.

  Brent Chapman

I enjoyed learning about their deliberate architectural choice to keep their central service in a single AZ. This incident highlighted a need for a fast failover plan.

  Coinbase

I like the balance between ensuring 99.99% reliability and designing their product to encourage customers to use their platform in a way that effectively manages the 0.01% case.

Reliability is a customer experience problem

  Mike Fisher — incident.io

I’m not gonna spoil this one for you by writing a summary. Just read it, trust me.

  Lorin Hochstein