I’ve shared this article before, but it’s so critical that it’s time to give it another read. MTTR is a statistically useless metric, and by using it, we will draw faulty conclusions and potentially take harmful actions. Courtney Nash does a really great job of laying out the science in an easy-to-understand way.

  Courtney Nash — Resilience in Software Foundation / The VOID

I like the analogy here: when we say people are components in or sociotechnical systems, system diagrams are like a form of cache.

  Clint Byrum

From Werner Vogels’s intro to this article:

Andy takes us through S3’s evolution from simple object store to sophisticated data platform, illustrating how customer feedback has shaped every aspect of the service. It’s a fascinating look at how we maintain simplicity even as systems scale to handle hundreds of trillions of objects.

  Andy Warfield — Amazon

Instead of a traditional Cost/Performance/Reliability trade-off, this article argues that serverless presents a tradeoff of Cost, Performance, and Complexity.

  Luc van Donkersgoed

Google uses System Theoretic Process Analysis to identify problems in their systems. They found that the most effective way to spread adoption of STPA was to build their own training program.

  Garrett Holthaus — Google

So far, I’m liking this new post series from Nextdoor about their efforts to scale their datastore. Here’s the first installment, about the things they’ve tried up to now.

I’ll share the rest of the series as I work my way through them.

  Slava Markeyev — Nextdoor

Wow, I had no idea EBS volumes failed this often!

  Nick Van Wiggeren — PlanetScale

No matter how bullet-proof you build the components of your system, the only way to make nines go up is to be ready to deal with the host of surprises that take them back down.

  Clint Byrum

Here’s an example of a really great application of bloom filters, in which speed is key and a slight risk of false is acceptable.

  Alex Gardiner — Klaviyo

This fun video gives us a small glimpse into the world of traffic light controllers, and more importantly, what makes them reliable. There’s also a longer video that goes deeper into why a Raspberry Pi isn’t up to the job.

  Traffic Light Doctor

Here’s an overview of several options to scale Prometheus beyond a single instance, including a handy table of features and functionality.

  Gaurav Maheshwari

A nice guide for using incident analysis in your home lab setup, plus a write-up for an incident experienced by the author.

  Barush Mendez

A highly detailed explanation of Paxos with diagrams and a model in FizzBee.

  Lorin Hochstein

I’ve boiled my frustration down to three problems:

1. No one agrees on what “microservice” means.
2. Microservices conversations are abstract, with little tie-in to real business goals
3. Adopting microservices without changing your organisation is pointless.

  Ian Miell — Container Solutions

It’s been awhile since we’ve seen any updates from the LFI folks, but here’s a brand new home for the community. I’ve bought my membership.

I like this article’s measured approach to anomaly detection and other AIOps features. Will it work? With your data?

  Jacek Migdal — Quesma

A structured approach to system design includes defining the problem, scope, tenets, risks, assumptions, and architecture choices.

I like how this article follows the process it lays out by writing an example design for a distributed search engine.

  Nikunj Agarwal — DZone

A mental model to detect and prevent optimizing the wrong thing, at the wrong time, or for the wrong reasons

This is the first time I’ve seen premature optimization dissected in this way, and I really like this model.

  Alex Ewerlöf

My favorite part of this podcast episode is the discussion of the unintended consequences of automation and “humans-are-better-at/machines-are-better-at” oversimplification. The transcript is great in case you’re not able to listen.

  Shane Hastie, with guest Courtney Nash — InfoQ

What role is an AI tool going to play in your sociotechnical system? This article gives you 12 insightful questions that will help guide your approach.

  Fred Hebert — Honeycomb

As long as there’s at least one HDD ‘tape’ filesystem mounted, you can count them, but once there are none, the result of counting them is not 0 but nothing.

And “nothing” doesn’t cause an alert. Oops!

  Chris Siebenmann

Just because Google, Amazon, or Facebook does it doesn’t mean you should. Here are four ‘best practices’ of the hyperscalers you have permission to ignore.

  Matt Asay — InfoWorld

An introduction to distributed transactions using the Saga pattern, including pros and cons and two approaches for implementing sagas.

  Sid — Scalable Thread

Here’s an argument against real-world “war rooms” for incident response, including a great incident story as an example.

I can’t imagine doing that kind of multi-window parallel investigation stuff on a teeny little laptop screen with people right next to me on either side

  rachelbythebay

This one includes a list of responsibilities a lead incident responder has and another list of things they should delegate.

Incident lead isn’t an extra job that you do “on top of” engineering. It’s the main job.

  r/devoopseng — Reddit r/sre

Scaling Elasticsearch requires balancing sharding, query performance, and memory tuning for optimal efficiency in high-traffic, real-time applications.

   Vivek Kumar — DZone

An incident report from the vault, along with its accompanying blog post, involving a rare but serious kernel freeze on GCP.

  Jake Cooper — Railway

Let’s discuss logging – unstructured, structured and canonical log lines – what they are and what value they bring to your production systems.

This one includes an example of implementing a logging system in an example project.

  Obakeng Mosadi

This article aims to answer one question: How can Redis be used as a primary database for complex applications that need to store data in multiple formats?

It covers persistence and scaling options, including Redis Enterprise’s built-in CRDTs.

   Mohammed Talib

In this blog post we’re going to explore how the hung task warning works, why it happens, whether it is a bug in the Linux kernel or application itself, and whether it is worth monitoring at all.

  Oxana Kharitonova and Jesper Brouer — Cloudflare

This post discusses key preconditions for building resilience, including resources, flexibility, expertise, diversity, and coordination.

  Lorin Hochstein

So the main problem with blameful postmortems is not the blame. It’s the very idea that particular decisions can be categorically unsafe.

  u/devoopseng — Reddit r/sre

This may be the shortest article I’ve ever linked to here, but it’ll make you think.

  Dean Wilson

If you use SLOs at all levels in your system, a failure of a core part (like the DB) may page multiple teams. This article offers strategies to handle this better.

  Fred Hebert — Honeycomb