The author of this one draws a line between their two interests of formal methods and resilience engineering, and I’m so here for it.

  Lorin Hochstein

In this part of the Scaling Nextdoor’s Datastores blog series, we’ll explore how the Core-Services team at Nextdoor serializes database data for caching while ensuring forward and backward compatibility between the cache and application code.

  Ronak Shah — Nextdoor

MySQL’s ALTER TABLE INPLACE has limitations and downsides, and INSTANT does too, as explained in this article.

  Shlomi Noach — Planetscale

If you have multiple different types of work in your system, a queue per type of work may be a good choice.

Bonus(?): includes a bathroom-based analogy.

  Marc Brooker

One Lambda function per URL path? Or a monolithic function that handles multiple paths? There are benefits and drawbacks to each.

  Yan Cui

Published on April 1.

The truth is, many incidents move faster when there’s executive oversight — a sense of urgency, pressure, and someone repeatedly asking, “What’s the ETA?”

  Chris Evans — incident.io

  This article is published by my sponsor, incident.io, but their sponsorship did not influence its inclusion in this issue.

I’m seeing a lot of echoes of Bainbridge’s Ironies of Automation in this article about AIOps and AI tooling. If AI handles most coding and incidents, how will humans handle the outliers?

  Hamed Silatani — Uptime Labs

I wasn’t able to make it, so I really appreciate this recap. Sounds like SRECon was, unsurprisingly, heavily focused on AI this time around.

  Niall Murphy

An SRE thinks about the meaning of “sociotechnical”:

From an SRE perspective, it means that when we’re looking at a piece of software, we can’t just factor out the human decisions that happen both in its operation and usage, but also in its development.

  Clint Byrum

This one is about the difficulties they had with database read replicas that led to developers mostly just sending reads to the primary. They came up with a pretty neat solution to automatically send read queries to the replica when possible.

In case you missed it, here’s part 1.

  Tushar Singla — Nextdoor

This well-thought-out article starts with a solid critique of Five Whys, illustrated with example scenarios. The author then explains why they prefer open-ended questions.

  Hamed Silatani

Spurred by a conversation with engineers, the author of this article explains what retries, backoff, and jitter can fix, and more importantly, when they won’t help.

  Tejas Ghadge — The New Stack

This is a juicy one, involving a routine credential roll gone bad, resulting in an outage in Cloudflare’s R2 service.

  Phillip Jones — Cloudflare

In this series of posts, we illustrate design considerations for a database system throttler, whose purpose is to keep the database system healthy overall. We discuss choice of metrics, granularity, behavior, impact, prioritization, and other topics.

Part 2 is here and part 3 is here.

  Shlomi Noach — Planetscale

I hadn’t heard the term “lurking variable” before, but I definitely know the concept. This article is a must-read for anyone troubleshooting tricky problems in production, and especially for earlier-career folks developing their skills.

  Teiva Harsanyi — The Coder Cafe

This article gives 4 strategies to better handle situations when database queries need to join across data residing in separate shards.

   Baskar Sikkayan — DZone

I’ve shared this article before, but it’s so critical that it’s time to give it another read. MTTR is a statistically useless metric, and by using it, we will draw faulty conclusions and potentially take harmful actions. Courtney Nash does a really great job of laying out the science in an easy-to-understand way.

  Courtney Nash — Resilience in Software Foundation / The VOID

I like the analogy here: when we say people are components in or sociotechnical systems, system diagrams are like a form of cache.

  Clint Byrum

From Werner Vogels’s intro to this article:

Andy takes us through S3’s evolution from simple object store to sophisticated data platform, illustrating how customer feedback has shaped every aspect of the service. It’s a fascinating look at how we maintain simplicity even as systems scale to handle hundreds of trillions of objects.

  Andy Warfield — Amazon

Instead of a traditional Cost/Performance/Reliability trade-off, this article argues that serverless presents a tradeoff of Cost, Performance, and Complexity.

  Luc van Donkersgoed

Google uses System Theoretic Process Analysis to identify problems in their systems. They found that the most effective way to spread adoption of STPA was to build their own training program.

  Garrett Holthaus — Google

So far, I’m liking this new post series from Nextdoor about their efforts to scale their datastore. Here’s the first installment, about the things they’ve tried up to now.

I’ll share the rest of the series as I work my way through them.

  Slava Markeyev — Nextdoor

Wow, I had no idea EBS volumes failed this often!

  Nick Van Wiggeren — PlanetScale

No matter how bullet-proof you build the components of your system, the only way to make nines go up is to be ready to deal with the host of surprises that take them back down.

  Clint Byrum

Here’s an example of a really great application of bloom filters, in which speed is key and a slight risk of false is acceptable.

  Alex Gardiner — Klaviyo

This fun video gives us a small glimpse into the world of traffic light controllers, and more importantly, what makes them reliable. There’s also a longer video that goes deeper into why a Raspberry Pi isn’t up to the job.

  Traffic Light Doctor

Here’s an overview of several options to scale Prometheus beyond a single instance, including a handy table of features and functionality.

  Gaurav Maheshwari

A nice guide for using incident analysis in your home lab setup, plus a write-up for an incident experienced by the author.

  Barush Mendez

A highly detailed explanation of Paxos with diagrams and a model in FizzBee.

  Lorin Hochstein

I’ve boiled my frustration down to three problems:

1. No one agrees on what “microservice” means.
2. Microservices conversations are abstract, with little tie-in to real business goals
3. Adopting microservices without changing your organisation is pointless.

  Ian Miell — Container Solutions

It’s been awhile since we’ve seen any updates from the LFI folks, but here’s a brand new home for the community. I’ve bought my membership.

I like this article’s measured approach to anomaly detection and other AIOps features. Will it work? With your data?

  Jacek Migdal — Quesma

A structured approach to system design includes defining the problem, scope, tenets, risks, assumptions, and architecture choices.

I like how this article follows the process it lays out by writing an example design for a distributed search engine.

  Nikunj Agarwal — DZone

A mental model to detect and prevent optimizing the wrong thing, at the wrong time, or for the wrong reasons

This is the first time I’ve seen premature optimization dissected in this way, and I really like this model.

  Alex Ewerlöf

My favorite part of this podcast episode is the discussion of the unintended consequences of automation and “humans-are-better-at/machines-are-better-at” oversimplification. The transcript is great in case you’re not able to listen.

  Shane Hastie, with guest Courtney Nash — InfoQ

What role is an AI tool going to play in your sociotechnical system? This article gives you 12 insightful questions that will help guide your approach.

  Fred Hebert — Honeycomb

As long as there’s at least one HDD ‘tape’ filesystem mounted, you can count them, but once there are none, the result of counting them is not 0 but nothing.

And “nothing” doesn’t cause an alert. Oops!

  Chris Siebenmann