A message from our sponsor, Incident.io:
AI made shipping code faster. Now reliability is the bottleneck.
Hear how Netflix and OpenAI handle that challenge at SEV0, the reliability conference in SF and London.
Free for SREWeekly readers with code SREWEEKLY.
Today, I learned that “hot take” does not mean what I thought it meant. Specifically, it does not mean “telling it like it is and not holding back”. My deepest apologies for last week’s issue, Alex.
Alex Ewerlöf
Grey failure, feedback loops, retry storms, and a mis-aligned safeguard, complexity, plus more — there’s a lot to dig into in this incident.
Lorin Hochstein
They tested using Jev for SRE tasks paired only with a deterministic scaffolding, not an LLM.
Yiming Su, Saad Mohammad Rafid Pial, Jackson Clark, and Tianyin Xu — SREGym
A case that AI doesn’t quietly swap out human work one-for-one, it reshuffles our roles into something harder. I see a strong thread of the Ironies of Automation throughout this one.
John Allspaw — Adaptive Capacity Labs
A short interview with a seasoned incident commander. My favorite part was question 3.
Simona Ursan — Uptime Labs
We’re going to do this via a metric that nobody’s ever heard of before, called “Mean Time to Stop Sucking” (MTSS). Just like in last week’s post, I’m going to argue that, despite the name, this is a real metric that you ought to be tracking in your autoscaler, and then explain why nobody does.
drmorr — ACRL
How we capture real production database traffic at Airbnb and replay it offline to load-test, plan capacity, and de-risk upgrades.
Zuofei Wang
Alert on what matters to your customers, not what you have easy metrics for. Reading this reminds me that I should set up a weekly review of on-call alerts to see which ones were actionable.
Brent Chapman