SRE Weekly Issue #543

A message from our sponsor, Incident.io:

Your agent is only as useful as the tools it can reach, and only as safe as the limits you set.

Join incident.io’s engineering team for an Extensions showcase. We’ll build a skill live, watch it run, and safely deploy.

Register Now →

The Resilience in Software Foundation is launching a series on key concepts to understand in resilience engineering. There’s so much to understand in the field that it can feel overwhelming. Each short article explains a concept with links on where to read more.

The first article gives four different ways to think about resilience.

I won’t automatically include every article in the series here in the newsletter, but I’ll showcase the ones I find especially interesting to help keep the series at top of mind.

  Resilience in Software Foundation

The eMudhra CA key incident report is out, and it’s full of systemic issues and non-technical motivators. Here’s Lorin’s take on it.

  Lorin Hochstein

A well-functioning autoscaler will minimize application churn without preventing the necessary scaling behavior your platform needs to serve your customers.

  Ian O’Gorman — Applied Computing Research Labs

Some of the best ICs I’ve worked with were project managers, customer support representatives, and salespeople. Their perspective can be incredibly valuable and make them more effective at Incident Command than engineers.

  Brent Chapman

Not only do they use verified formal specifications for a production service, they have monitoring and alerting for deviations from the model. Cool!

  Nicolas Couvrat — Datadog

Neki’s Admin system does not know whether the primary has failed or is merely unreachable from where Admin happens to run.

  Shlomi Noach — Planetscale

Four common areas that incident analysis can fall short, how to identify them, and what you can do. Tag yourself, I’m #4, or at least some organizations I’ve been in are.

  Oakley Hall

A chlorine dosing system failed, 15 guests were injured, and this week’s Outlook outage was a contributing factor. This is our regular reminder that the systems we operate can be safety-critical even if we don’t realize it.

  KarloBag.eu