A message from our sponsor, Rootly:
When incidents impact your customers, failing to communicate with them effectively can erode trust even further and compound an already difficult situation. Learn the essentials of customer-facing incident communication in Rootly’s latest blog post:
https://rootly.com/blog/the-medium-is-the-message-how-to-master-the-most-essential-incident-communication-channels
Articles
This article makes a cool analogy between designing systems to operate well under unexpected load and designing socio-technical systems that operate well when the people are surprised by what the system is doing.
Lorin Hochstein
If you need to create SLAs, this article has some solid advice on how to go about it — and what to avoid.
incident.io
If Prometheus can’t scrape your service, an alert can get resolved incorrectly — and that can happen exactly when your service is failing!
Chris Siebenmann
A really nifty three-part exploration of action items in the aftermath of an incidents. Rather than consider cost/benefit, this article series proposes that we think about the likelihood of an action item being completed.
J. Paul Reed
Yes, as it turns out — and these folks have the receipts (along with some theories as to why).
Colin Bartlett
The “wow” moment in this article is under the heading, “What can we learn from creative desperation?”
Eric Dobbs — Learning From Incidents
Before explaining how they set up their on-call, these folks share why they avoided it in the early stages of their startup, and what made them finally take the plunge.
Dustin Brown — DoltHub
For the good of the profession, the SRE community still needs to coalesce around more consistent job ladders, expectations, and competencies.
Code Reliant
Honeycomb had their worst incident ever at the end of July, and in their characteristic style, they’ve posted an incredibly detailed analysis of what happened — and that’s just the blog post. Then you can click through for a 17-page PDF with lots more detail.
Fred Hebert — Honeycomb
Full disclosure: Honeycomb is my employer.
