SRE Weekly Issue #542

A message from our sponsor, Archestra:

AI chaos in your org? Archestra is an open-source enterprise AI platform used by Fortune 50s that organizes your AI: LLM proxy, MCP gateway, skill catalog, feature factory, governance, observability and deterministic guardrails.

⇨ Book a call with the core team.

I’ve seen this so many times. If declaring an incident gets in the way of responding to it, something’s wrong.

  Hugo van der Horst

They have a maximum concurrency in Lambda, and they want to ensure that no one misbehaving client can exhaust it on its own.

  Daniil Romashov

This one’s really interesting. The PLC refused to enable generators because the standby utility power was live. However, that standby utility power couldn’t be used because the breaker was purposefully opened during maintenance. I’m sure thorough failure testing was completed, it’s just that this kind of scenario hadn’t occurred to them.

  Google

The fire service doesn’t track mean time to put out a fire.  Why should we measure our MTTR?

  Brent Chapman

Wow. Between this, the difficulty in reverting back to standard coreutils, and ubuntu’s schedule in publishing a fix for CVE-2026-98432, I’m having some second thoughts on Ubuntu. And no, Chris, I don’t think you should be expected to file a bug report the sudo-rs stuff.

  Chris Siebenmann

It’s like the security equivalent of Lorin’s law: a new process put in place with the goal of improving security (for a subset of users) itself was a the core of a security incident.

I’m really looking forward to uMudhra’s incident report, due tomorrow.

  Lorin Hochstein

An overview of queues and what they’re good for with a reliability engineering lens.

  Alex Ewerlöf

Runbook as hand-off mechanism. Runbook as a pointer to who knows more. Runbook as an embodiment of hard-earned lessons from the past. This one has me thinking about runbooks in new ways.

  Tim Irving