CrowdStrike released a lot more discussion about what happened widetailth their bad deployment, and yet there’s still a frustrating lack of detail on the actual cause of the blue screens.

  CrowdStrike

A story of how properly positioned rationales can be powerful enough to prevent prod incidents

And a great place to put that rationale is in your git commit, says this article.

  Jean-Mark Wright

Need to choose between Redis and Memcached? This one’s for you, with a qualitative comparison and relative performance numbers.

  Rahul Chandel

How do you promote interactions between fans without exploding your system with n-squared worth of messages, where n is the number of users?

  Matthew O’Riordan — Ably

If you want to convert to serverless, don’t switch to microservices or change your datastore at the same time, argues this article.

  Yan Cui

It’s all about the owl’s butt (and sending 4 million push notifications in 5 seconds).

  Zhen Zhou — InfoQ

Here’s the second part of the article I wrote on my recent project at work, taking down a full AZ in production.

  Lex Neva — Honeycomb

  Full disclosure: Honeycomb is my employer.

The big news this week, of course, is the CrowdStrike-related series of outages in airports, banks, and many other businesses. Here’s their statement on the situation.

Rumor has it that Southwest Airlines survived because they run Windows 3.1. Well, that’s one way to do it.

  CrowdStrike

It’s time for Catchpoint’s annual SRE survey again! We get a lot of interesting information about SRE trends from this, so it’d be great if you could take a moment to fill it out.

Note, usually I try to avoid giving you “utm” stuff in links, but this link is specifically set up to track whether folks come from SRE Weekly, so I left it in this time.

  Catchpoint

Queues have a cost, as this article explains.

  Jean-Mark Wright

I wrote this article about an exciting project I led recently: taking down an entire availability zone in production to test reliability. Part 2 is due out next week!

  Lex Neva — Honeycomb

  Full disclosure: Honeycomb is my employer.

Deletion protection: it can really save you!

  Andre Newman — Gremlin

A thorough overview of Netflix’s architecture, with focus on data stores, content processing, billing, and the CDN, among other topics.

   Rahul Shivalkar — ClickIT

This article compares the terms “degradation”, “disruption”, and “service outage” through the lens of service levels.

  Alex Ewerlöf

Their workload involved writing many small objects but reading very few. By batching many writes into a single object in S3, they saved a ton of money, and now they’re open sourcing their solution.

  Pablo Matias Gomez — Embrace

This article covers five skills:

1. Ability to Lead
2. Taking Charge in Critical Situations
3. Expressing Opinions in a Non-Conflicting Way
4. Leading Initiatives for Continuous Improvement
5. Building and Maintaining Relationships

  Prabesh

I was pretty dubious most of the way through this article — until I realized it was a story about why this solution didn’t work for them. Now it’s an interesting read about Python and exercising restraint in complexity.

  Jean-Mark Wright

Meta is training an LLM to suggest commits that may have caused a given incident, and its suggestions are right 42% of the time.

  Diana Hsu, Michael Neu, Mohamed Farrag, and Rahul Kindi — Meta

Percentiles, because when your math(s) teacher told you you’d use math all the time when you grew up, they were right! This article does a great job of explaining percentiles if you’re having trouble wrapping your mind around them.

  Alex Ewerlöf

Netflix designed their load shedding system to efficiently drop the requests that don’t matter as much and prioritize what users really care about.

  Anirudh Mendiratta, Kevin Wang, Joey Lynch, Javier Fernandez-Ivern, and Benjamin Fedorka — Netflix

This article illustrates cascading delays in microservices and describes three techniques for dealing with them: timeouts, retries, and circuit breakers.

  Jean-Mark Wright

Cloudflare’s public DNS resolver had an outage due to a (probably accidental?) BGP hijack. 1.1.1.1 is a common address used internally for testing routing, so it’s easy to understand how an accidental route leak happened.

   Bryton Herdes, Mingwei Zhang, and Tanner Ryan — Cloudflare

Here’s a new post about durability and write-ahead logs. Write-ahead logs are used almost everywhere. But to build an intuition for why, it is helpful to imagine what you would do without a WAL.

  Phil Eaton

In this debugging story, an engineer wielded SystemTap to figure out why a Kafka broker was doing a ridiculous amount of reads.

  Terra Field — Honeycomb

  Full disclosure: Honeycomb is my employer.

A concise breakdown of the math involved in getting that extra nine of reliability.

It all boils down to creating the SLOs and requirements to keep your users happy, but nothing more. Unnecessary reliability is a high cost.

  Thomas Stringer

If you’re looking to advance in SRE, this article has some examples of the skills and experience you should aim for.

  Prabesh

Will Gallego shows us a way of thinking that helps turn “should haves” into deeper understanding of our sociotechnical systems.

  Will Gallego

Some words of wisdom I came across this week around startups choosing not to work on scalability too early.

   Vassil Popovski

Some commenters in this reddit thread are saying it’s easier to be called an SRE, but what does it mean? Some say SRE has gotten easier, and some say it’s gotten harder. What do you think?

  u/sreiously and others — reddit

The full report isn’t available yet (and may not ever be?) but this executive summary has a lot of juicy bits about the major 2022 Rogers internet and emergency service outage in Canada.

  Xona Partners, Inc.

The Rogers report executive summary includes some blamey and blame-adjacent language, and this analysis does a good job of calling it out and suggesting ways to recast it.

  Lorin Hochstein

The Rogers outage report executive summary indicates that truly out-of-band network management access may have made recovery easier. What exactly is involved in setting that up?

  Chris Siebenmann

This is a really thorny one. As individual subprocesses started infinitely looping, their system shifted load to other datacenters, masking the problem. A coinciding failure in the load shifting system made things even more interesting.

  Lloyd Wallis, Julien Desgats, and Manish Arora — Cloudflare

A great discussion of where dashboards fall short and what we should look for instead.

  Adam Kinniburgh — SquaredUp

Read how we have significantly improved the ability of our monolith to correctly and fully process pushes from our users.

  Will Haltom — GitHub

Timing things to happen at specific intervals is yet another way that we collectively find out that dealing with time is a hard problem.

This article illustrates the subtle but important pitfalls in trying to create a system that does something on a strict interval.

  rachelbythebay

This article reads more like a case study. The author gave a prompt to three different LLMs and actually tested the Terraform config it produced.

  Mike Vanbuskirk — Terrateam

When your pub/sub system can have a million subscribers, even something mundane as notifying about subscriber counts requires careful thought.

  Ashmeet Singh — Pusher

To me, this concept comes up over and over in SRE, and it’s a core part of SLOs.

  Juraj Masar — BetterStack

In this blog post, we’ll dive deep into the technical aspects of feature flags and feature management, exploring how they can be leveraged by SREs to enable progressive delivery, improve system resilience, and optimize the user experience.

  Hope Lynch — CloudBees

This week’s Mentour Pilot video covers an accident that involved an inaccurate flight simulator. I wasn’t familiar with the term “negative training” before, but now I’m going to be keeping an eye out for it in the systems I manage!

  Mentour Pilot