Why don’t we look for the root cause of a successful outcome?

  Hamed Silatani — Uptime Labs

They took a great deal of care to avoid the potential pitfalls of using an LLM in this way, and they share a lot of detail about the steps they took.

  Tran Le, Till Pieper, and Gillian McGarvey — Datadog

After dealing with a late-night outage with surprisingly small impact, I got thinking about how you would know if you were working too hard to guarantee uptime.

  Tom Elliott

In this article, learn how the 4 R’s — robust architecture, resumability, recoverability, and redundancy — enhance reliability in AI and ML data pipelines.

   Sidhant bendre — DZone

In this article, I’ll delve into the challenges we encountered and the strategies we employed to manage operator upgrades for stateful workloads like Elasticsearch. Additionally, I’ll detail how we modified the ECK [Elastic Cloud on Kubernetes] operator to facilitate a more resilient side-by-side upgrade process.

  Abhishek Munagekar — Mercari

In this piece, I’ll delve into four macro challenges facing observability today, explore strategies that are emerging across the industry to address them, and offer my perspective on the trajectory of this crucial domain in the year to come.

  Andrew Mallaband

A deep-dive into a pretty nifty system for enumerating and provisioning a rack of servers, complete with PXE-based Debian headless installation using an auto-generated preseed file. It also uses Claude to figure out what state a server is in from a screenshot obtained from the BMC.

  Charith Amarasinghe — Railway

Koreo is a new open source tool for orchestrating Kubernetes infrastructure at a higher level than standard tools like Helm.

Koreo is a fairly complex tool, so it can be difficult to quickly grasp just what exactly it is, what problems it’s designed to solve, and how it compares to other, similar tools. In this post, I want to dive into these topics and also discuss the original motivation behind Koreo.

  Tyler Treat

This one is about understanding how work actually happens in our sociotechnical systems (versus how we imagine it). This has implications for how we learn from incidents and how we design corrective actions.

  Lorin Hochstein

The myth is:

The underlying and often unexamined assumption for the benefits of automation is the notion that computers/machines are better at some tasks, and humans are better at a different, non-overlapping set of tasks.

This article lays out several pitfalls to this approach, with references.

  Courtney Nash

Wow, I seriously love this one. It’s written in an a very approachable style that’s easy to understand from the outside. It lays a series of cringe-worthy contributing factors that could happen to any of us, making them a great learning opportunity.

  Spotify

This is the first time I’ve come across the term “grounding” in incident response, and I like it!

At the core of our vision lies the principle of grounding, drawn from safety-critical systems like aviation and the fire service industries. Grounding is the process of maintaining a shared understanding among team members throughout the course of an incident.

  Uptime Labs

I really like the idea of using formal modeling on distributed systems. Datadog explains how they did it when building a new message queuing service.

  Arun Parthiban, Sesh Nalla, and Cecilia Wat-Kim

I found this to be a really useful primer on the new EU AI regulation. It does transition into a sales pitch toward the end, but the pre-pitch content is substantial.

  Chris Evans — incident.io

A classic example of Lorin’s Law: work intended to improve reliability was at the heart of this incident.

  Railway

Feature flags are incredibly useful, but they have some gotchas too.

  Tom Elliott

More potential problems to watch out for with feature flags, but this one ends by emphasizing that feature flags are still an important tool. Bonus points for a Knight Capital Incident mention.

  Ian Vanagas

I haven’t seen this level of detail in an article on anomaly detection in quite awhile. Still, the math is very approachable even if you slept through stats class.

  Ivan Shubin — Booking.com

TL;DR: The Power of Knowledge Overlap in Incident Response

There’s an anecdote in this one that’s really making me think.

  Hamed Silatani — Uptime Labs

One of the criticisms leveled at resilience engineering is that the insights that the field generates aren’t actionable […]

This article argues that we still need the unactionable but good models, otherwise we’ll get actionable but wrong models.

  Lorin Hochstein

Datadog has put a lot of thought and effort into managing their massive Kafka workload. My favorite part of this article was the bit about accidentally zip-bombing themselves with highly compressible data.

  Guillaume Bort — Datadog

This one covers four techniques for rerouting customer traffic after a region failure using AWS’s Route 53… themed after the TV show The Good Place. It’s been quite awhile since I watched the show, but I still found the article pretty useful.

  Seth Elliot — Arpio

This article asks what we’re really looking to get by defining an incident severity scale, and proposes an alternative scale based on incident complexity.

  Dan Slimmon

I love this idea of tracking configuration changes as observability data. I’ve been through plenty of incidents in which I wish I had it.

  Yevgeny Pats — CloudQuery

A short and sweet article packed with some useful nuggets. My favorite is the section near the end on timeouts.

  Hemant Burman — Insights

This is a truly outstanding article about blameless incident analysis! Beyond just “why”, it covers many of the pitfalls that trip people up when they try to enact a blameless culture, including questions about accountability.

  fgj

Here’s a good reminder that resilience in our systems is all about the humans.

  Stuart Rimell

This article outlines WarpStream’s solution to a common problem in systems based on shared storage (like S3): cleaning up objects that are no longer needed, at scale.

  Richard Artoul — WarpStream

I love learning how companies structure their on-call rota. My favorite part of this one is the emphasis on keeping the manager in the rota as a feedback mechanism.

  Laura de Vesine and David Lentz — Datadog

These folks continuously detect drift by running terraform plan and alerting on changes that have no corresponding commit in git.

   Yugandhar Suthari

It’s a troubleshooting story having nothing to do with tech, but the technique used can easily apply to your next incident.

  Paige Cruz

Some examples you may not have thought of that can lead to Terraform drift, along with an exploration of the problems drift can bring.

  Saijal Shrivastava — Razorpay

Railway had an outage this week related to their control plane database, and they shared this write-up.

  Ray Chen — Railway

In this final installment of the Scaling Nextdoor’s Datastores blog series, we detail how the Core-Services team at Nextdoor solved cache consistency challenges as part of a holistic approach to improve our database and cache scalability and usability.

I really enjoyed this whole series. Thanks, Nextdoor folks!

  Slava Markeyev — Nextdoor

These folks analyzed a non-production incident like it was production, including retrospective analysis and lessons learned. Best part: they share the juicy details with us!

  Joe Mckevitt — UptimeLabs

This one goes over several different models you can use to implement on-call compensation, with pros and cons for each.

  Constant Fischer — PagerDuty

This article shows that MySQL’s CATS algorithm offers only a small performance gain over FIFO once deadlock logging interference is removed.

My jaw involuntarily opened when I saw the graph after they commented out the logging print statements.

   Bin Wang — DZone

In this article, I’ll walk you through how we implemented chaos engineering across our stack using Chaos Toolkit, Chaos Monkey, and Istio — with hands-on examples for Java and Node.js. If you’re exploring ways to strengthen system resilience, this guide is packed with practical insights you can apply today.

The author does not appear to have a tie to Istio. This article has a ton of code snippets to help you get started.

   Prabhu Chinnasamy — DZone

In this blog, we’ll look at three important facts about serverless reliability that teams often overlook. We’ll explain what they are, what the risks are of not addressing them, and how you can make your serverless applications more fault-tolerant.

1. Serverless architectures don’t guarantee reliability.
2. You do have control over serverless reliability.
3. Serverless reliability practices can benefit all platforms, not just serverless platforms.

  Andre Newman — Gremlin

This Golang debugging story is a really satisfying read.

The heap profiles were very effective at telling us the allocation sites of live objects, but provided no insights into why specific objects were being retained.

  Ella Chao — WarpStream

Zoom had an outage this week when its domain zoom.us was temporarily blocked at the TLD level due to a miscommunication between its registrar and the TLD.

  Zoom