Most fascinating air incident report I’ve seen in awhile! The pilots deviated from the non-normal checklist, and it immediately made me think of runbooks. On the one hand, you want the runbook to be simple and easy to handle in an incident. On the other hand, it can be very useful to tell the operator why they should do something.

  Mentour Pilot

With their claimed 14.5% of all websites depending on Cloudflare’s DNS, they had to be super careful with this migration. Lots of good stuff in here including:

• replacing direct DB access by multiple services with an API
• keeping the old and new DB in sync
• ensuring both forward and reverse migration were possible in case of rollback

  Alex Fattouche and Corey Horton — Cloudflare

I didn’t get to experience the value of a good tracing tool until recently in my career, and I didn’t understand the hype. If you’re in the same boat, this article may help you understand the value of tracing.

  Sam Starling — incident.io

About a year ago, Honeycomb git rid of incident severity levels in favor of incident types, which are purposefully not sortable. Here’s how their experiment has gone so far.

  Fred Hebert — Honeycomb

  Full disclosure: Honeycomb is my employer.

Is Service Level Indicator (SLI) the same as Key Performance Indicator (KPI)?

There’s a really cool framing in there: KPIs are moonshots, so we aim high and rarely hit all of them, while with SLOs, we under-promise and over-deliver.

  Alex Ewerlöf

A fun dive into some unix/linux internals with nine different methods to run a program with timeouts and retries. If you have a soft spot in your heart for signals and system calls, this one’s for you.

  Philippe Gaultier

Cosmos DB is Azure’s answer to Amazon’s DynamoDB. This article gives a nice overview and compares it to various other data stores to help you decide whether it’s right for your use case.

  Adam Gordon Bell — Pulumi

An engineer at Mercari shares their plan for migrating to their new payment system in this five-part article series, all of which are published now. They created their design after reading 80(!) similar articles from folks at other companies.

  resotto — Mercari

If you’re heading to KubeCon this week, here are some talks to consider.

  JJ Tang — Rootly

This article shows you how to manage Postgres indexes: when you need one, what type of index to choose, and how to set it up.

  Milly Leadley — incident.io

It’s neat that Cloudflare can see evidence of a BGP route leak in a third party that affected OVHcloud.

  Bryton Herdes, David Belson, and Tanner Ryan — Cloudflare

In this post, we’ll explore how we leveraged Temporal’s own capabilities to expand our infrastructure from AWS to Google Cloud, the challenges we faced along the way, and how we solved them using cloud-agnostic workflows.

  Raphaël Beamonte — Temporal

This deeply opinionated piece advocates against the “Architect” role, at least in certain forms. Among other problems, the Architect role breaks ownership models and impedes others.

  Alex Ewerlöf

These folks were using Kubernetes for their product that provides hosted developer environments (build systems, toolchains, and the like). While they directly acknowledge that their use case is not the same as common production environments, I still found it pretty interesting to learn about the problems they ran into that ultimately caused them to find another platform.

  Christian Weichel and Alejandro de Brito Fontes — Gitpod

This extensive guide shows you how to build an LLM-based agent to assist with incident response. It includes python code snippets and shows you how to provide the LLM agent with documentation and access to external data sources like PagerDuty.

  Eric Abruzzese — Aptible

A primer on the Swiss Cheese Model for modeling how accidents happen. I especially like the section at the end that suggests more nuanced thinking.

  Jonathan Cheyne — Johnson Winter Slattery

This new series seems promising! I won’t link to every article in the series here, but if you’re an early SRE, the intro-level articles published so far in this series are definitely worth a read.

Today, I’m thrilled to announce an ambitious project that’s been in the works for some time: “52 Weeks of SRE” – a comprehensive, year-long deep dive into the world of Site Reliability Engineering.

  J. Pereira

Adevinta shifted from Kubernetes’s cluster autoscaler to AWS’s Karpenter. The change brought huge advantages that they discuss in detail, along with a few challenges and pitfalls they needed to overcome.

  Tanat Lokejaroenlarb — Adevinta

An adventure in adopting an open source firmware for Baseboard Management Controllers, including fixing a few bugs themselves.

  Nnamdi Ajah, Ryan Chow, and Giovanni Pereira Zantedeschi — Cloudflare

[…] an overview of methods like TCP FastOpen, TLSv1.3, 0-RTT, and HTTP/3 to reduce handshake delays and improve server response times in secure environments.

   Maksim Kupriianov — DZone

This article includes general tips and a specific rubric you can follow to decide when to choose a larger or smaller RDS instance type.

  Prabesh

It turns out that a lot of the lessons that Mike Massimino learned as an astronaut apply very well to incident management.

  Eric Silberstein — Klaviyo

Solving IP exhaustion in EKS: Avoiding a network outage by implementing custom networking

  Fabián Sellés Rosa — Adevinta

By leveraging proportional–integral–derivative (PID) controllers, Robinhood can now more quickly and effectively manage load imbalances.

This was my first introduction to PID controllers. Neat!

  Yi-Shu Tai — Dropbox

Through an allegory about an imaginary knob to adjust between risk-avoidance and speed, Lorin Hochstein shows us that these trade-offs are being made, just implicitly.

  Lorin Hochstein

They traded their monolith for microservices in a quest for scalability, but they got complexity along with it.

   Jennifer Riggins — The New Stack

Here’s a great summary of the difference between mutable and immutable infrastructure, including a detailed analysis of the pros and cons of each.

   Josephine Eskaline Joyce and Umar Ali — DZone

An introduction to incident severity and SEV1 incidents, along with how to respond to them, how to prevent them, and how to learn from them.

  Kate Bernacchi-Sass — incident.io

Long-running spans can be difficult to deal with, but fortunately Hazel Weakly is here with an explanation and some tips.

  Hazel Weakly — The New Stack

Here’s a debugging odyssey for a truly gnarly Jupyter Notebook problem that caused slowness in very specific and (seemingly unrelated) circumstances.

  Hechao Li and Marcelo Mayworm — Netflix

Beyond just “What went well?” in an incident writeup, Lorin urges examining our incidents to see what they can tell us about how work gets done and what adaptations people have made in our systems.

  Lorin Hochstein

A huge primer on wide events in observability: what they are, how to implement them, how to use them, and a ton of examples of the kinds of fields you might want to include in your events.

  Jeremy Morrell

  Full disclosure: Honeycomb, my employer, is mentioned.

The 2024 DORA Report is out, and the folks at Rootly have some thoughts on the interesting bits for SREs including AI, platform engineering, and burnout.

  Jorge Lainfiesta — Rootly

There are quite a few pitfalls waiting for you if you try to implement SLOs for your mobile app. This article explains and offers strategies.

   Virna Sekuj — The New Stack

Blamelessness in incident retrospectives can be a difficult concept to truly internalize. This article describes 3 common “failure modes”, that is, ways in which organizations struggle with blamelessness.

  Tom Elliott — The Friday Deploy

Cloudflare spends a lot of time thinking about cooling, and it’s fascinating. I didn’t realize that spinning a fan faster consumed so much more energy!

  Leslye Paniagua — Cloudflare

Explore the pitfalls associated with the excessive creation of microservices, insights on their causes, implications, and potential strategies for mitigation.

   Sumit Kumar — DZone

Netflix stores a truly obscene number of events, each of which has a timestamp and a set of key-value pairs. This article goes into a ton of detail on how they built their system.

  Rajiv Shringi, Vinay Chella, Kaidan Fullerton, Oleksii Tkachuk, and Joey Lynch — Netflix

A fun debugging story for a confusing crash bug, in which they found 6 other related bugs along the way.

  Brett Wines — Slack

My favorite one is about the principle “You Ain’t Gonna Need It”:

The flip side of YAGNI, however, is that at some point you might actually need it.

  Luc van Donkersgoed

When you create an index on multiple columns in Postgres, you’ll need to be sure that the order of the fields in the index allows it to be applied to your queries, as these folks learned.

  Jean-Mark Wright