This article makes a cool analogy between designing systems to operate well under unexpected load and designing socio-technical systems that operate well when the people are surprised by what the system is doing.
If you need to create SLAs, this article has some solid advice on how to go about it — and what to avoid.
If Prometheus can’t scrape your service, an alert can get resolved incorrectly — and that can happen exactly when your service is failing!
A really nifty three-part exploration of action items in the aftermath of an incidents. Rather than consider cost/benefit, this article series proposes that we think about the likelihood of an action item being completed.
J. Paul Reed
Yes, as it turns out — and these folks have the receipts (along with some theories as to why).
The “wow” moment in this article is under the heading, “What can we learn from creative desperation?”
Eric Dobbs — Learning From Incidents
Before explaining how they set up their on-call, these folks share why they avoided it in the early stages of their startup, and what made them finally take the plunge.
Dustin Brown — DoltHub
For the good of the profession, the SRE community still needs to coalesce around more consistent job ladders, expectations, and competencies.
Honeycomb had their worst incident ever at the end of July, and in their characteristic style, they’ve posted an incredibly detailed analysis of what happened — and that’s just the blog post. Then you can click through for a 17-page PDF with lots more detail.
Fred Hebert — Honeycomb
Full disclosure: Honeycomb is my employer.