Here’s a wonderful tribute to the many ways Dr. Cook has advanced our field and others.

  John Allspaw — Adaptive Capacity Labs

This seems like a fitting time to feature Dr. Cook’s seminal treatise here again.

  Dr. Richard Cook

A good argument could be made either way, but what really caught my eye was this (emphasis mine):

Responding to incidents should distract as few people as reasonably possible. Organisations should be shooting for minimum viable participation, whilst still responding effectively, to allow them to retain focus.

  Chris Evans — incident.io

Noticing a correlation between the adoption of SRE and cloud repatriation (moving apps out of the cloud), the author of this article asks, is there causation?

  Lori Macvittie — Devops.com

I like the line this article draws between incident retrospectives and developing a PRR process, and also the emphasis on psychological safety.

Incidents reveal what your organization is good at and what needs improvement in your PRR processes.

  Nora Jones — Jeli

Aperture is a new open source tool helps you prevent cascading failures using load-shedding and rate limiting.

BONUS CONTENT: Here‘s their article explaining how it works.

  FluxNinja

This one advocates for looking beyond “root cause” when analyzing an incident, and instead finding Themes and Takeaways.

If it can be solved with a pull request it’s not a takeaway.

  Vanessa Huerta Granda — Jeli

In this juicy incident, the Incident Commander’s intimate knowledge of a similar failure mode fixated incident response away from the true cause.

  Fred Hebert — Honeycomb

[…] the more we normalize lower-impact incidents, the more confidence and experience we build for Sev1 situations.

  Dan Condomitti — The New Stack

Want to compensate folks extra for on-call work? This tool connects to PagerDuty to do all the heavy lifting for you.

  Lawrence Jones — incident.io

This Reddit post in r/sre has some really great stories in the comments.

  various users — Reddit

Along with the “why”, this article also goes into the “how”.

  Martha Lambert — incident.io

Early in my career, I had to write a raw IP packet generator to reproduce a DoS attack so that I could mitigate it. It’s fun!

  Julia Evans

In an incident in July, a cloud provider change broke provisioning for new Codespaces VMs, taking down the service.

  Jakub Oleksy — GitHub

Put Safety First and Minimize
the 12 Common Causes of Mistakes
in the Aviation Workplace

  FAA (US’s Federal Aviation Administration)

Every one of these 10 items is enough reason to read this article! This makes me want to go investigate some incidents right now.

  Fischer Jemison — Jeli

Slack shares with us in great detail why they use circuit breakers and how they rolled them out.

  Frank Chen — Slack

My favorite part of this one is the section on expectations. We need to socialize this to help reduce the pressure on folks going on call for the first time.

  Prakya Vasudevan — Squadcast

Status pages are marketing material. Prove me wrong.

  Ellen Steinke — Metrist

incidents have unusually high information density compared with day-to-day work, and they enable you to piggy-back on the experience of others

  Lisa Karlin Curtis — incident.io

These folks realized that they had two different use cases for the same data, real-time transactions and batch processing. Rather than try to find one DB that could support both, they fork two copies of the data.

  Xi Chen and Siliang Cao — Grab

It’s all about gathering enough information that you can ask new questions when something goes wrong, rather than being stuck with only answers to the questions you thought to ask in advance.

  Charity Majors

They needed the speed of local ephemeral SSDs but the reliability of network-based persistent disks. The solution: a linux MD option to mirror but prefer to read from the local disks. Neat!

  Glen Oakley — Discord

OS upgrades can be risky. LinkedIn developed a system to unify OS upgrade procedures and make them much less risky.

  Hengyang Hu, Dinesh Dhakal, and Kalyanasundaram Somasundaram — LinkedIn

In this article, I will introduce several improvements being made by the Microservices SRE Team, embedded with other teams.

  MizumotoShota — Mercari

What really stood out to me in this article is the Service Info section. A dashboard will quickly atrophy and lose its meaning without an explanation of what it’s for.

  Ali Sattari

When things go wrong, who is in charge? And what does it feel like to do that role?

This is a summary of a forum discussion about incident command, in case you don’t have time to listen to the whole thing.

  Emily Arnott — Blameless

Complex systems are weird, and a traditional deterministic view such as in older ITIL iterations doesn’t capture the situation. We need to evolve our practices.

  Jon Stevens-Hall

How can you design and interpret metrics for systems optimized for latency or throughput?

  Dan Slimmon

You can optimize for latency or throughput in a given system, but not both, since the two are directly at odds.

  Dan Slimmon

I really like that “Missing” section in their incident retrospective template. Gotta be careful with “Missed” though, that sounds like it could slide toward blame.

  Varun Achar — Razorpay

“Unreasonable” is a great way to avoid learning from an incident:

Labeling the responders actions as unreasonable enables us to explain away the failures in the law enforcement response as deficiencies with the individual responders.

  Lorin Hochstein

The author of this post doesn’t argue the fact that Fastly is clearly a single point of failure for many of their customers. But does that really matter?

  Jon Stevens-Hall
Full disclosure: Fastly, my employer, is mentioned.

Small problems can pile up unnoticed and interact weirdly to make a Big Problem that is incredibly hard to untangle. Maybe we should hunt down the small problems before they have a chance to trigger a Big one.

  Dan Slimmon

Apologizing for bugs encourages a lot of problematic thought patterns, much in the same way as blaming people for incidents.

  Dan Slimmon