My coworkers referred to a system “going metastable”, and when I asked what that was, they pointed me to this awesome paper.

Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is `removed.

  Nathan Bronson, Aleksey Charapko, Abutalib Aghayev, and Timothy Zhu

Honeycomb posted this incident report involving a service hitting the open file descriptors limit.

  Honeycomb
  Full disclosure: Honeycomb is my employer.

Lots of interesting answers to this one, especially when someone uttered the phrase:

engineers should not be on call

  u/infomaniac89 and others — reddit

A misbehaving internal Google service overloaded Cloud Filestore, exceeding its global request limit and effectively DoSing customers.

  Google

An in-depth look at how Adobe improved its on-call experience. They used a deliberate plan to change their team’s on-call habits for the better.

  Bianca Costache — Adobe

This one contains an interesting observation: they found that outages caused by a cloud providers take longer to solve.

  Jeff Martens — Metrist

Even if you don’t agree with all of their reasons, it’s definitely worth thinking about.

  Danny Martinez — incident.io

This one covers common reliability risks in APIs and techniques for mitigating them.

  Utsav Shah

The evolution beyond separate Dev and Ops teams continues. This article traces the path through DevOps and into platform-focused teams.

  Charity Majors — Honeycomb
  Full disclosure: Honeycomb is my employer.

This one’s from a couple years ago and covers 3 main themes the author saw at SRECon Americas 2020. Fascinating topics include providing context for newbies, learning from incidents, and rethinking the incident command system.

  Taylor Barnett — Transposit

On September 8, Honeycomb had a major outage in data ingestion, and they’ve posted this preliminary report, “pending an in-depth incident review in the upcoming weeks”.

BONUS CONTENT: Another outage report from a different outage the next day.

  Honeycomb
Full disclosure: Honeycomb is my employer.

This is neat! Someone posted a day in their life as an actual SRE, and a bunch of commenters followed suit.

  Various commenters — Reddit

Some big names in SRE got together to talk about how to know when your system is broken. Listen to the recording or read this excellent summary that goes in depth on grey failures and more.

  Emily Arnott — Blameless

To better scale our systems, our infrastructure and product teams got together and decided to make these optimizations: reduce database loads, conduct load tests and size the demand and prioritize critical flows.

…and sharding.

  Robinhood

A major incident went poorly, and that catalyzed investment in developing a new incident response system. They worked to transition from swarming to Incident Command.

  Vikrant Saini — Razorpay

I love this part:

[…] if you have to deploy your microservices in a certain order, they’re not really microservices.

  Cortex

This one had an interesting interplay of contributing factors.

  Heroku

Here’s a wonderful tribute to the many ways Dr. Cook has advanced our field and others.

  John Allspaw — Adaptive Capacity Labs

This seems like a fitting time to feature Dr. Cook’s seminal treatise here again.

  Dr. Richard Cook

A good argument could be made either way, but what really caught my eye was this (emphasis mine):

Responding to incidents should distract as few people as reasonably possible. Organisations should be shooting for minimum viable participation, whilst still responding effectively, to allow them to retain focus.

  Chris Evans — incident.io

Noticing a correlation between the adoption of SRE and cloud repatriation (moving apps out of the cloud), the author of this article asks, is there causation?

  Lori Macvittie — Devops.com

I like the line this article draws between incident retrospectives and developing a PRR process, and also the emphasis on psychological safety.

Incidents reveal what your organization is good at and what needs improvement in your PRR processes.

  Nora Jones — Jeli

Aperture is a new open source tool helps you prevent cascading failures using load-shedding and rate limiting.

BONUS CONTENT: Here‘s their article explaining how it works.

  FluxNinja

This one advocates for looking beyond “root cause” when analyzing an incident, and instead finding Themes and Takeaways.

If it can be solved with a pull request it’s not a takeaway.

  Vanessa Huerta Granda — Jeli

In this juicy incident, the Incident Commander’s intimate knowledge of a similar failure mode fixated incident response away from the true cause.

  Fred Hebert — Honeycomb

[…] the more we normalize lower-impact incidents, the more confidence and experience we build for Sev1 situations.

  Dan Condomitti — The New Stack

Want to compensate folks extra for on-call work? This tool connects to PagerDuty to do all the heavy lifting for you.

  Lawrence Jones — incident.io

This Reddit post in r/sre has some really great stories in the comments.

  various users — Reddit

Along with the “why”, this article also goes into the “how”.

  Martha Lambert — incident.io

Early in my career, I had to write a raw IP packet generator to reproduce a DoS attack so that I could mitigate it. It’s fun!

  Julia Evans

In an incident in July, a cloud provider change broke provisioning for new Codespaces VMs, taking down the service.

  Jakub Oleksy — GitHub

Put Safety First and Minimize
the 12 Common Causes of Mistakes
in the Aviation Workplace

  FAA (US’s Federal Aviation Administration)

Every one of these 10 items is enough reason to read this article! This makes me want to go investigate some incidents right now.

  Fischer Jemison — Jeli

Slack shares with us in great detail why they use circuit breakers and how they rolled them out.

  Frank Chen — Slack

My favorite part of this one is the section on expectations. We need to socialize this to help reduce the pressure on folks going on call for the first time.

  Prakya Vasudevan — Squadcast

Status pages are marketing material. Prove me wrong.

  Ellen Steinke — Metrist

incidents have unusually high information density compared with day-to-day work, and they enable you to piggy-back on the experience of others

  Lisa Karlin Curtis — incident.io

These folks realized that they had two different use cases for the same data, real-time transactions and batch processing. Rather than try to find one DB that could support both, they fork two copies of the data.

  Xi Chen and Siliang Cao — Grab

It’s all about gathering enough information that you can ask new questions when something goes wrong, rather than being stuck with only answers to the questions you thought to ask in advance.

  Charity Majors

They needed the speed of local ephemeral SSDs but the reliability of network-based persistent disks. The solution: a linux MD option to mirror but prefer to read from the local disks. Neat!

  Glen Oakley — Discord

OS upgrades can be risky. LinkedIn developed a system to unify OS upgrade procedures and make them much less risky.

  Hengyang Hu, Dinesh Dhakal, and Kalyanasundaram Somasundaram — LinkedIn