Three days ago, PagerDuty had a major incident, severely impacting incident creation, notifications, and more. Linked above is a discussion on reddit’s r/sre with lots of takes on how folks deal with this kind of thing.

  u/Secret-Menu-2121 and others

It’s not telepathy; it’s about building common ground. This article explains what that means and the components that comprise common ground in an incident.

  Stuart Rimell — Uptime Labs

An introduction to database connection pooling in general, and RDS proxy in specific, complete with a Terraform snippet.

  David Kraytsberg — Klaviyo

This article explores the difference between simple and easy, their relation to complexity, and the effect of production pressure.

  Lorin Hochstein

What does “High Availability” actually mean? It turns out that it can mean different things to different people, and it’s important to look deeper.

  Teiva Harsanyi — The Coder Cafe

This short but sweet untitled LinkedIn post goes into the importance of understanding the entire context rather than focusing on an individual’s mistakes or omissions.

  Ron Gantt

Whether you’re just getting started implementing SLIs and SLOs or you’re a veteran, you’ll want to read this one. It charts the progress of organizations as they successively refine and mature their SLIs, and more importantly, it explains why the later stages matter.

  Alex Ewerlöf

This 2-part episode of The VOID Podcast is just awesome, and well worth a listen. The conversation is framed as a retrospective of a simulated incident, with a high level of expertise and experience in the incident participants and the retrospective facilitator. I have a lot to think about, especially the discussion of overload and the four ways people react to it.

  Courtney Nash — The VOID Podcast, with guests Sarah Butt, Eric Dobbs, Alex Elman, and Hamed Silatani

Discover how tail sampling in OpenTelemetry enhances observability, reduces costs, and captures critical traces for faster detection and smarter system monitoring.

   Rishab Jolly — DZone

Datadog has evolved their time series storage through five generations before, and now they’re on the sixth. Click through to find out what motivated each change and what’s different this time around.

  Khayyam Guliyev, Duarte Nunes, Ming Chen, and Justin Jaffray — Datadog

Meta uses a tool to automatically estimate the risk level of a code change. They’ve used this to reduce the use of code freezes.

  Meta

The authors of Catchpoint’s SRE Report look back at their analysis and predictions related to AIOps, compared to how things are unfolding now.

  Leo Vasiliou and Denton Chikura — The New Stack

I love the approach and the level of detail in this article. They gave four LLMs access to observability data in a simulated infrastructure and asked them to troubleshoot a problem. It’s super useful to see the actual results from the LLMs.

  Lionel Palacin and Al Brown — ClickHouse

Uptime Labs goes meta by sharing the details of an incident they experienced last month, involving runaway creation of dynamic queues in RabbitMQ.

  Joe Mckevitt — Uptime Labs

I’m pretty impressed: Cloudflare published this article with a ton of detail on an incident, the day after it happened. A surge of traffic overloaded Cloudflare’s Data Center Internet connect links to AWS’s us-east-1 region.

  David Tuber, Emily Music, Bryton Herdes — Cloudflare

Catchpoint’s yearly survey is live! This time, they’ll plant a tree for each of the first 2000 respondents.

  Catchpoint

If you’re looking to build a status page, this article is for you. It gives reviews of 10 status pages and sums it up with a list of things to consider as you design yours.

  Sara Miteva — Checkly

The GCP outage on June 12 hit Cloudflare hard, and they’ve responded by redesigning their Workers KV service to eliminate the dependency on a third party cloud.

   Alex Robinson and Tyson Trautmann — Cloudflare

I found the bit about Google’s historical reasons for SRE especially interesting.

  Dave O’Connor

There’s a fascinating point in this article explaining why “eventual consistency” may sound entirely different to German speakers. It continues on to a really good explanation of what eventual consistency actually means.

  Uwe Friedrichsen

This article introduces SLI Compass, a 2D mental model to help you:

• Quickly assess the signal/noise ratio of existing SLIs
• Evaluate SLIs based on their cost and complexity
• Set a direction for improving the quality of existing SLIs at a reasonable ROI

  Alex Ewerlöf

This is a really interesting failure mode for an endpoint monitoring provider.

  Tomas Koprusak — UptimeRobot

As we learn advanced resilience engineering concepts, this article recommends that we take a balanced approach in how we try to change existing practices.

I can confidently say that when an executive leader wants to be talking about quality of service for your customers, the last thing they want to hear about is academic papers and Monte Carlo simulations.

  Michelle Casey — Resilience in Software Foundation

I know you probably know all about how hashing works, but this one’s still worth a read. The article includes interactive demonstrations and clearly presents concepts to help you understand how hashing function performance is evaluated.

  Sam Rose

Pulled from the Internet Archive, here’s a story of how the now-defunct Parse rewrote their Ruby on Rails API in Golang, significantly improving reliability.

  Charity Majors

We are sharing methodologies we deploy at various scales for detecting SDC [Silent Data Corruption] across our AI and non-AI infrastructure to help ensure the reliability of AI training and inference workloads across Meta.

  Harish Dattatraya Dixit and Sriram Sankar — Meta

As monday.com broke their monolith up into microservices, their number of databases expanded too. To have a chance of managing all of them, they shifted from DBA practices to DBRE.

  Mateusz Wojciechowski — monday.com

Airbnb runs a large-scale database on Kubernetes. They have various techniques to deal with the ephemerality of pods and the risks inherent in cluster upgrades.

  Artem Danilov — Airbnb

The author of this article brings us along as they do a very thorough evaluation of K8sGPT, showing us what it can do and some ways in which it can fall short.

  Evgeny Torin — Palark

What is good incident communication? This article draws on theory from Herbert Clark’s Joint Action Ladder to help us evaluate and strengthen communication.

  Stuart Rimell — Uptime Labs

A story of the failure of a pumped energy storage facility, involving all of our favorite features like complex contributing factors, work-as-done vs work-as-designed, and early warning signs only obvious in hindsight. As a bonus, no one was killed.

  Practical Engineering

Nebula‘s streaming service has a surprisingly write-heavy workflow owing to storing bookmarks of the latest point a given user has watched in a video. That makes scaling an interesting challenge.

   Sam Rose — Nebula

I love the debugging technique they used: kill processes one at a time until performance improves.

  Samson Hu, Shashank Tavildar, Eric Kalkanger, and Hunter Gatewood — Pinterest

This article is about finding the balance between having enough process to ensure incident response goes smoothly, and having so much process that incident responders are unable to adapt to unexpected situations.

  Brandon Chalk — Rootly

This article presents two case studies of dialog during incidents along with analysis of each. How does your own analysis compare?

  Hamed Silatani — Uptime Labs

They realized that a single alert can’t catch both a sudden AC failure and an AC that becomes slowly but steadily overwhelmed.

  Chris Siebenmann

Thoughts on migrations as a significant source of reliability risk.

[…] engineering organizations at tech companies need to make migrations a part of their core competency, rather than seeing them as one-off chores.

  Lorin Hochstein

An incorrect physical disconnection was made to the active network switch serving our control plane, rather than the redundant unit scheduled for removal.

This reminds me of wrong-side surgery incidents and aircraft pilots shutting off the good engine when one fails.

  Google