When writing about an incident, it’s important to skillfully show the reader how the participants’ understanding of the situation evolved.

Lorin Hochstein

This is a summary of Bainbridge’s seminal paper, and I really love where Adrian Colyer goes with it.

One example I found myself thinking about while reading through the paper does have a human precedence though: self-driving cars.

Adrian Colyer — The Morning Paper (summary)

Bainbridge — Automatica (original paper)

I have to admit, it is brilliant. Why add the risk (and latency) of a centralized configuration repository service when a local DB on each host will do?

Rick Branson — Segment

This one covers a lot. My favorite parts:

• Permissive failure — if Netflix’s subscriber information service is down they just show videos for free, favoring reliability over correctness.
• Human attention span — if it takes 10 minutes to see if your changes broke production, you’re likely to wander off and work on something else.

Adrian Cockcroft

The author guides you through the moment they began to truly understand what observability is all about. Worth reading even if you’re already quite familiar with the concept.

Sanjeev Sharma

This article describes our work with NS1 to optimize our intelligent DNS-based global load balancing for corner cases that we uncovered while improving our point of presence (PoP) selection automation for our edge network.

Grab uses bulkheading to prevent localized demand spikes from affecting the service for customers elsewhere. The notable part is that they shed load they can’t satisfy anyway, due to a limited supply of available vehicles.

Corey Scott — Grab

Looking from multiple perspectives is incredibly important to effectively learn from an incident. Equally true for asking what went right.

Subbu Allamaraju

Failure to anticipate and design for
the new challenges that are certain to arise following periods of technology change leads
to automation surprises when advocates are surprised by negative unintended consequences that offset apparent benefits

Thanks to Greg Burek for this one.

David Woods — Ohio State University

Start the year off with this refreshingly deep dive into how variable-argument functions in C work.

Jan Schaumann

Think you know how to write files safely, say with fsync() or something? Think again.

In conclusion, computers don’t work

Dan Luu

The logical argument goes like this: if incidents in your system each had a single root cause, that implies a level of brittleness that would preclude your company running successfully at all.

Lorin Hochstein

Once a system reaches a certain level of reliability, most major incidents will involve:

• A manual intervention that was intended to mitigate a minor incident, or
• Unexpected behavior of a subsystem whose primary purpose was to improve reliability

Lorin Hochstein

Confirmation bias can lead us to reinforce an incorrect mental model through spurious correlations.

Thai Wood — Resilience Roundup (summary)
Dennis Bernard, David Greathead, and Gordon Baxter — International Journal of Human Computer Studies (original paper)

In this post, I’ll recap his talk, sharing the journey that led them to build GoAlert, the problems they’ve solved, and how they use GoAlert with Sensu Go to simplify monitoring and reduce alert fatigue.

Anna MacLachlan — Sensu (recap)
Adam Westman — Target (talk)

Verbose debug logging + feature flagging = a way to investigate unknown unknowns in your system.

Will Sargent

Domino model, Swiss Cheese model, stand aside. The Gamma Knife model is a nifty analogy for contributing factors.

Lorin Hochstein

Lots of great tips here for how to make things easier on yourself when you’re paged. Pave the way for your 3 am self to get things fixed and get back to sleep as soon as possible.

Katie McLaughlin (Sysadvent day 21)

Ooh, a new SRE podcast! PagerDuty started things up with 4 episodes right out of the gate.

Introducing “Page It To The Limit,” a new podcast by the Community team here at PagerDuty that discusses what it means to operate software in production.

Wow, I love the idea of this shadowing program. The author discusses incidents they saw and 5 things they learned while shadowing.

Tristan Read — GitLab

“in aviation safety, it’s like we’ve been trying to learn about marriage by only studying divorce.”

Kristy Kiernan — Forbes

Use the right tool for the job, not the coolest one.

Mattias Geniar

In line with last week’s article on patience by Will Gallego, this one emphasizes the importance of continued learning about resilience engineering.

Lorin Hochstein

Here are some really thought-provoking tips on how (and why) to write an effective post-incident analysis.

Lorin Hochstein

To get better at avoiding or mitigating future incidents, you need to understand the conditions that enabled past incidents to occur. Counterfactual reasoning is actively harmful for this, because it circumvents inquiry into those conditions.

Lorin Hochstein

Some great observations and questions related to the Cloudflare outage in July.

Lorin Hochstein

Sometimes, things are off, and you just know an incident is brewing. What is this skill, and how can we learn it?

Silvia Botros — Learning From Incidents