This article explores the advantages of powering SLOs with observability data.

  Pierre Tessier — Honeycomb
  Full disclosure: Honeycomb is my employer.

As the James Webb Space Telescope moves into normal operations, there are more great SRE lessons to be learned.

  Jennifer Riggins — The New Stack

During 5 years of experience as an SRE, the author of this article gathered a set of best practice patterns for software development and operation, which they share with us.

  brandon willett

How Airbnb built a persistent, high availability and low latency key-value storage engine for accessing derived data from offline and streaming events.

  Chandramouli Rangarajan, Shouyan Guo, Yuxi Jin — Airbnb

By owning and reporting MTTR, teams have no choice but to be accountable for the reliability of the code they write. This dramatically changes the culture of engineering.

  Sidu Ponnappa — Last9

I learned about plan continuation bias while reading this air accident report, and I’m certain I’ve experienced this during incidents I’ve been involved in.

  Admiral Cloudberg

As a television broadcaster, how do I ensure that my channels are playing out the right thing for my viewers?

This is SRE applied to tv broadcasting: they replaced human monitoring of screens with an automated system.

  Jeremy Blythe — evertz.io
  Full disclosure: Honeycomb, my employer, is mentioned.

An interview with an engineer about on-call practices, training folks for on-call, and chaos engineering.

  Elena Boroda — Fiberplane

SRE: totally defined. Time for a reorg, and with a catchy tune!

  Forrest Brazeal

Great advice for incident response, backed up by real-world anecdotes.

  Audrey Simonne — DZone

There’s a lot to learn from in this air accident. A chilling example: several quirks of the plane’s automation combined to effectively tell the pilot to continue pushing the plane to stall.

  Admiral Cloudberg

When sharding a database, if transactions can span shards, then it can be very difficult to reason about the system’s maximum throughput.

For example, splitting a single-node database in half could lead to worse performance than the original system.

  Marc Brooker

Through Ubuntu’s unattended-upgrades system, a systemd update was installed that broke systemd-resolved, which in turn broke GitHub Codespaces. The systemd bug report they link to is also well worth a read.

  Jakub Oleksy — GitHub

Why not?

we’re, unfortunately, too good at explaining away failures without making any changes to our priors.

  Lorin Hochstein

My coworkers referred to a system “going metastable”, and when I asked what that was, they pointed me to this awesome paper.

Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is `removed.

  Nathan Bronson, Aleksey Charapko, Abutalib Aghayev, and Timothy Zhu

Honeycomb posted this incident report involving a service hitting the open file descriptors limit.

  Honeycomb
  Full disclosure: Honeycomb is my employer.

Lots of interesting answers to this one, especially when someone uttered the phrase:

engineers should not be on call

  u/infomaniac89 and others — reddit

A misbehaving internal Google service overloaded Cloud Filestore, exceeding its global request limit and effectively DoSing customers.

  Google

An in-depth look at how Adobe improved its on-call experience. They used a deliberate plan to change their team’s on-call habits for the better.

  Bianca Costache — Adobe

This one contains an interesting observation: they found that outages caused by a cloud providers take longer to solve.

  Jeff Martens — Metrist

Even if you don’t agree with all of their reasons, it’s definitely worth thinking about.

  Danny Martinez — incident.io

This one covers common reliability risks in APIs and techniques for mitigating them.

  Utsav Shah

The evolution beyond separate Dev and Ops teams continues. This article traces the path through DevOps and into platform-focused teams.

  Charity Majors — Honeycomb
  Full disclosure: Honeycomb is my employer.

This one’s from a couple years ago and covers 3 main themes the author saw at SRECon Americas 2020. Fascinating topics include providing context for newbies, learning from incidents, and rethinking the incident command system.

  Taylor Barnett — Transposit

On September 8, Honeycomb had a major outage in data ingestion, and they’ve posted this preliminary report, “pending an in-depth incident review in the upcoming weeks”.

BONUS CONTENT: Another outage report from a different outage the next day.

  Honeycomb
Full disclosure: Honeycomb is my employer.

This is neat! Someone posted a day in their life as an actual SRE, and a bunch of commenters followed suit.

  Various commenters — Reddit

Some big names in SRE got together to talk about how to know when your system is broken. Listen to the recording or read this excellent summary that goes in depth on grey failures and more.

  Emily Arnott — Blameless

To better scale our systems, our infrastructure and product teams got together and decided to make these optimizations: reduce database loads, conduct load tests and size the demand and prioritize critical flows.

…and sharding.

  Robinhood

A major incident went poorly, and that catalyzed investment in developing a new incident response system. They worked to transition from swarming to Incident Command.

  Vikrant Saini — Razorpay

I love this part:

[…] if you have to deploy your microservices in a certain order, they’re not really microservices.

  Cortex

This one had an interesting interplay of contributing factors.

  Heroku

Here’s a wonderful tribute to the many ways Dr. Cook has advanced our field and others.

  John Allspaw — Adaptive Capacity Labs

This seems like a fitting time to feature Dr. Cook’s seminal treatise here again.

  Dr. Richard Cook

A good argument could be made either way, but what really caught my eye was this (emphasis mine):

Responding to incidents should distract as few people as reasonably possible. Organisations should be shooting for minimum viable participation, whilst still responding effectively, to allow them to retain focus.

  Chris Evans — incident.io

Noticing a correlation between the adoption of SRE and cloud repatriation (moving apps out of the cloud), the author of this article asks, is there causation?

  Lori Macvittie — Devops.com

I like the line this article draws between incident retrospectives and developing a PRR process, and also the emphasis on psychological safety.

Incidents reveal what your organization is good at and what needs improvement in your PRR processes.

  Nora Jones — Jeli

Aperture is a new open source tool helps you prevent cascading failures using load-shedding and rate limiting.

BONUS CONTENT: Here‘s their article explaining how it works.

  FluxNinja