SRE Weekly Issue #202

When writing about an incident, it’s important to skillfully show the reader how the participants’ understanding of the situation evolved.

Lorin Hochstein

This is a summary of Bainbridge’s seminal paper, and I really love where Adrian Colyer goes with it.

One example I found myself thinking about while reading through the paper does have a human precedence though: self-driving cars.

Adrian Colyer — The Morning Paper (summary)

Bainbridge — Automatica (original paper)

I have to admit, it is brilliant. Why add the risk (and latency) of a centralized configuration repository service when a local DB on each host will do?

Rick Branson — Segment

This one covers a lot. My favorite parts:

• Permissive failure — if Netflix’s subscriber information service is down they just show videos for free, favoring reliability over correctness.
• Human attention span — if it takes 10 minutes to see if your changes broke production, you’re likely to wander off and work on something else.

Adrian Cockcroft

The author guides you through the moment they began to truly understand what observability is all about. Worth reading even if you’re already quite familiar with the concept.

Sanjeev Sharma

This article describes our work with NS1 to optimize our intelligent DNS-based global load balancing for corner cases that we uncovered while improving our point of presence (PoP) selection automation for our edge network.

Grab uses bulkheading to prevent localized demand spikes from affecting the service for customers elsewhere. The notable part is that they shed load they can’t satisfy anyway, due to a limited supply of available vehicles.

Corey Scott — Grab