SRE Weekly Issue #543

A message from our sponsor, Incident.io:

Your agent is only as useful as the tools it can reach, and only as safe as the limits you set.

Join incident.io’s engineering team for an Extensions showcase. We’ll build a skill live, watch it run, and safely deploy.

Register Now →

The Resilience in Software Foundation is launching a series on key concepts to understand in resilience engineering. There’s so much to understand in the field that it can feel overwhelming. Each short article explains a concept with links on where to read more.

The first article gives four different ways to think about resilience.

I won’t automatically include every article in the series here in the newsletter, but I’ll showcase the ones I find especially interesting to help keep the series at top of mind.

  Resilience in Software Foundation

The eMudhra CA key incident report is out, and it’s full of systemic issues and non-technical motivators. Here’s Lorin’s take on it.

  Lorin Hochstein

A well-functioning autoscaler will minimize application churn without preventing the necessary scaling behavior your platform needs to serve your customers.

  Ian O’Gorman — Applied Computing Research Labs

Some of the best ICs I’ve worked with were project managers, customer support representatives, and salespeople. Their perspective can be incredibly valuable and make them more effective at Incident Command than engineers.

  Brent Chapman

Not only do they use verified formal specifications for a production service, they have monitoring and alerting for deviations from the model. Cool!

  Nicolas Couvrat — Datadog

Neki’s Admin system does not know whether the primary has failed or is merely unreachable from where Admin happens to run.

  Shlomi Noach — Planetscale

Four common areas that incident analysis can fall short, how to identify them, and what you can do. Tag yourself, I’m #4, or at least some organizations I’ve been in are.

  Oakley Hall

A chlorine dosing system failed, 15 guests were injured, and this week’s Outlook outage was a contributing factor. This is our regular reminder that the systems we operate can be safety-critical even if we don’t realize it.

  KarloBag.eu

SRE Weekly Issue #542

A message from our sponsor, Archestra:

AI chaos in your org? Archestra is an open-source enterprise AI platform used by Fortune 50s that organizes your AI: LLM proxy, MCP gateway, skill catalog, feature factory, governance, observability and deterministic guardrails.

⇨ Book a call with the core team.

I’ve seen this so many times. If declaring an incident gets in the way of responding to it, something’s wrong.

  Hugo van der Horst

They have a maximum concurrency in Lambda, and they want to ensure that no one misbehaving client can exhaust it on its own.

  Daniil Romashov

This one’s really interesting. The PLC refused to enable generators because the standby utility power was live. However, that standby utility power couldn’t be used because the breaker was purposefully opened during maintenance. I’m sure thorough failure testing was completed, it’s just that this kind of scenario hadn’t occurred to them.

  Google

The fire service doesn’t track mean time to put out a fire.  Why should we measure our MTTR?

  Brent Chapman

Wow. Between this, the difficulty in reverting back to standard coreutils, and ubuntu’s schedule in publishing a fix for CVE-2026-98432, I’m having some second thoughts on Ubuntu. And no, Chris, I don’t think you should be expected to file a bug report the sudo-rs stuff.

  Chris Siebenmann

It’s like the security equivalent of Lorin’s law: a new process put in place with the goal of improving security (for a subset of users) itself was a the core of a security incident.

I’m really looking forward to uMudhra’s incident report, due tomorrow.

  Lorin Hochstein

An overview of queues and what they’re good for with a reliability engineering lens.

  Alex Ewerlöf

Runbook as hand-off mechanism. Runbook as a pointer to who knows more. Runbook as an embodiment of hard-earned lessons from the past. This one has me thinking about runbooks in new ways.

  Tim Irving

SRE Weekly Issue #541

A message from our sponsor, incident.io:

Your agent is only as useful as the tools it can reach, and only as safe as the limits you set.

Join incident.io’s engineering team for an Extensions showcase. We’ll build a skill live, watch it run, and safely deploy.

Register Now →

Archaeological layers of P0 incident action items? There’s a better way.

  Brent Chapman

Wow, the original for this article was one I included all the way back in issue #16. I think I’ve quoted the concepts from that one a million times.

  J. Paul Reed

At incident.io, agents now go beyond coding their mobile app changes, they actually run the app in a simulator and provide visual evidence of the change for a human to review.

  Grayson Ho — incident.io

Getting Debezium to stream the first PostgreSQL change into Kafka is straightforward; keeping it correct through failover is the interesting part.

  Frane Jelavic — ShiftMag

This one shows the drawbacks of focusing on a root cause through a really cool lens: a Five Whys analysis of the sinking of the Titanic.

  Steve Mynett — Rootly

Giving an agent more context can make it less effective. Here’s Meta’s technique to mitigate that.

  Plawan Rath, Akhil Jindal and Ashwini Verma — Meta

Migrating the busiest 0.00001% of repositories to Azure took some careful work and consideration of bottlenecks.

  Eleanor White and Victoria Dye — GitHub

My favorite section is the one about achieving higher reliability by coordinating between their client library and the server side.

  Matt Hammond — Ably

SRE Weekly Issue #540

A message from our sponsor, Dragonfly:

Redis runs one core per process. Your server has 64. Dragonfly is a Redis/Valkey-compatible data store, rebuilt multi-threaded by an ex-AWS ElastiCache engineer:

  • 6M ops/sec on one node, no cluster
  • Drop-in: same clients, same commands
  • Instacart migrated and cut latency by 50% using 70% fewer nodes

Read how

Events like the recent compromise of CA signing keys are discussed out in the open, on Mozilla’s Bugzilla. Right now, you can watch the post-incident analysis as it unfolds.

  Mozilla

This article gives a really easy-to-follow explanation of the consistency issues you get when writing across two systems and how to solve them with an outbox pattern.

   Rahul Tewari — DZone

An explanation of why a service catalog is so important, especially during incidents.

Twenty minutes in, you haven’t started diagnosing anything. You’ve been doing archaeology.

Bonus: part two on keeping your service catalog up-to-date.

  Bill Duncan

The Firebase SDK caused app-side crashes due to a server-side data change. This article focuses on Google’s response, and I also recommend reading the writeup on the Firebase blog that Gergely links to.

  Gergely Orosz — The Pragmatic Engineer

LLM agents behave fundamentally differently from deterministic programs, and we can’t just expect them to work like a natural language programming language.

  Uwe Friedrichsen

This one delves into the design choices involved in running tons of little databases at the edge efficiently.

  David Laban and Lambros Petrou — Cloudflare

A seasoned emergency responder and computer incident management engineer shares a rare perspective on the similarities and differences between the two fields.

  Brent Chapman

Now I start a chat session with an AI model, give him a personality, most often it is an old and grumpy but well-mannered distinguished engineer…

  Hrvoje Rancic — ShiftMag

SRE Weekly Issue #539

A message from our sponsor, Incident.io:

AI made shipping code faster. Now reliability is the bottleneck.

Hear how Netflix and OpenAI handle that challenge at SEV0, the reliability conference in SF and London.

Free for SREWeekly readers with code SREWEEKLY.

Claim your free pass →

Today, I learned that “hot take” does not mean what I thought it meant. Specifically, it does not mean “telling it like it is and not holding back”. My deepest apologies for last week’s issue, Alex.

  Alex Ewerlöf

Grey failure, feedback loops, retry storms, and a mis-aligned safeguard, complexity, plus more — there’s a lot to dig into in this incident.

  Lorin Hochstein

They tested using Jev for SRE tasks paired only with a deterministic scaffolding, not an LLM.

  Yiming Su, Saad Mohammad Rafid Pial, Jackson Clark, and Tianyin Xu — SREGym

A case that AI doesn’t quietly swap out human work one-for-one, it reshuffles our roles into something harder. I see a strong thread of the Ironies of Automation throughout this one.

  John Allspaw — Adaptive Capacity Labs

A short interview with a seasoned incident commander. My favorite part was question 3.

  Simona Ursan — Uptime Labs

We’re going to do this via a metric that nobody’s ever heard of before, called “Mean Time to Stop Sucking” (MTSS). Just like in last week’s post, I’m going to argue that, despite the name, this is a real metric that you ought to be tracking in your autoscaler, and then explain why nobody does.

  drmorr — ACRL

How we capture real production database traffic at Airbnb and replay it offline to load-test, plan capacity, and de-risk upgrades.

  Zuofei Wang

Alert on what matters to your customers, not what you have easy metrics for. Reading this reminds me that I should set up a weekly review of on-call alerts to see which ones were actionable.

  Brent Chapman