Reading this article will teach you the math you need to know to build alerting that has a low false positive rate and why this is trickier than it may seem.

  Dan Slimmon

Cloudflare adapted a technique shared by PagerDuty to detect failed Kafka consumers and restart them.

   Chris Shepherd and Andrea Medda — Cloudflare

Gracefully shutting down is important, otherwise every deploy will result in client-facing errors.

  Srinavas — eightnoteight

There’s a wealth of lessons learned in this article. My favorite: idempotency was never part of the contract, but consumers nevertheless depended on it.

  Gergely Orosz — Pragmatic Engineer

Making our companies into High Reliability Organizations (HROs) rarely makes sense, but we can still learn useful skills and techniques from them. This article gives a good overview and analysis of HROs.

  Dominic Cooper — Safety & Health Practitioner

By “tiered”, this article means having discussions about reliability at three levels: the engineering team level, the director level, and the executive level.

  Ross Brodbeck

After explaining why deploys aren’t the right approach, this article proposes feature flags as a safer approach.

  Charity Majors — Honeycomb
  Full disclosure: Honeycomb is my employer.

In incident management as in so many areas, there’s the shiny work and the unglamorous but critical parts, and the latter often fall to women. This article seeks to reverse that trend by reminding us of the incredibly important glue work Women have been doing since the dawn of computing.

  Emily Arnott — Blameless

I love stories about applying IT incident response processes to non-IT incidents.

  Robert Ross — FireHydrant

Dear reader, perhaps you would enjoy reading this article on the many benefits of engineering blogs… then go write more great content and send me a link. :D

  New York Times — Jordan News

Okay, this isn’t exactly an SRE story, but it sounds really familiar. It’s a story of “user error” that’s really about designing systems to help users catch errors.

  Jakub Roztocil — httpie

nginx has a pretty nifty zero-downtime restart system, but it didn’t quite fit Cloudflare’s needs.

  Maciej Lechowski — Cloudflare

This article does a great job of summarizing SRECon Americas by pulling out five major themes that ran through multiple talks.

  Gavin Cahill — Gremlin

Building buy-in is everything.

[…] the key function of SRE being to help shape engineering’s perception of reality rather than act as a gatekeeper.

  Ross Brodbeck

By “FinOps”, they mean a team in your company dedicated to reducing cloud computing costs. Does that really help?

  Lydia Leong

[…] it is also possible to create incident writeups that engineers choose to read, that clearly describe and highlight difficult and poorly-understood aspects of our systems, and that become part of the organisation’s collective understanding.

  Laura Nolan — Container Solutions`

Years after we both started doing the newsletter thing, I finally sat down with Corey Quinn for an episode of his podcast. We talked about running newsletters, my other side project, and of course, reliability.

  Corey Quinn — Last Week In AWS

They take us from the requirements analysis all the way through implementation of a high-throughput data store based on CockroachDB.

  Chuanpin Zhu and Debalin Das — DoorDash

On March 14th, Reddit engineers upgraded a Kubernetes cluster from 1.23 to 1.24, and all hell broke loose. I admire their precision in being down for 100π minutes.

  Jayme Howard — Reddit

With a huge user-base of students and teachers, these folks upped their incident response game, and they share how.

  Nadinastiti and Estu Fardani — GovTech Edu

A lurking bug in redis-py allowed users to see one another’s data, and OpenAI took ChatGPT down to limit the damage.

  OpenAI

In Linux, source port allocation can be complex. This article shows why with a ton of code and tracing examples.

  Jakub Sitnicki — Cloudflare

The gap between “paying for peak” and “earning on average” is critical to understand how the economics of large-scale cloud systems differ from traditional single-tenant systems.

  Marc Brooker

A configuration error was masked because the app automatically fell back to the original configuration. The problem only surfaced when the service was redeployed.

  Heroku

Heresy! This article provides a counterpoint to many of the benefits of IaC. While IaC may still be the right answer, it’s not a slam dunk.

  Luke Shaughnessy

Short but sweet, this article outlines three focus areas that the author argues should be a part of any SRE role.

  Kyle Robertson

Way beyond just an intro to aperture, this article also covers microservice architecture failure modes, techniques used to avoid failures, and the weaknesses in those techniques.

  Cong Ma and Matt Ranney — Doordash

I’m including this here not just for the staff+ SREs out there. Many of these skills are important for SREs to develop much earlier than the Staff level, since our role can be so collaborative.

  Ryn Daniels — GitHub

I love that fully half of this article is about mentoring developing SREs in identifying and managing risk.

  Ross Brodbeck

Learn how the Honeycomb SRE team has structured its work, including a fully copy of the team charter.

  Fred Hebert — Honeycomb
  Full disclosure: Honeycomb is my employer and I am a member of the SRE team described in this article.

An intriguing approach: define technical debt as a risk, and manage it in much the same way that we handle reliability-related risks, with a “threat budget”.

  Jason Bloomberg — Intellyx

Instead, because our time and attention is limited, we have to get good at identifying cues to indicate that our models have gotten stale or are incorrect.

  Lorin Hochstein

Using a simulation, this article comes to the conclusion that a hybrid between FIFO and LIFO is better than picking just one.

   Eugene Retunsky — DZone

A super in-depth look at on-call compensation strategies. Includes a sampling of companies and how much they pay (if anything).

  Gergely Orosz — The Pragmatic Engineer

Husky uses a nifty sharding strategy where a customer’s shard allocation changes over time automatically based on load.

  Daniel Intskirveli — Datadog

This analogy goes far enough to even include rules. Anyone up for a round?

  Robert Ross

[…] in order to be truly great at being an SRE you will constantly need to understand how to work with people in the organization, how to set expectations and how to move the needle on people’s understanding of reliability.

  Ross Brodbeck

MongoDB -> Cassandra -> ScyllaDB. Storing a ton of stuff is hard.

  Bo Ingram — Discord

When designing complex technical systems, you should ask yourself, “how does the human operator fit into the picture”.

  Cursed Quail

It sounds like it was a great conference!

  Paige Cruz — Chronosphere

[…] complex systems don’t yield to analysis. We have to add another skill: sense-making.

  Jessica Kerr — Honeycomb
  Full disclosure: Honeycomb is my employer.