General

SRE Weekly Issue #419

A message from our sponsor, FireHydrant:

FireHydrant is now AI-powered for faster, smarter incidents! Power up your incidents with auto-generated real-time summaries, retrospectives, and status page updates. https://firehydrant.com/blog/ai-for-incident-management-is-here/

Our nine month journey to horizontally shard Figma’s Postgres stack, and the key to unlocking (nearly) infinite scalability.

Retrofitting sharding is a huge undertaking.

  Sammy Steele — Figma

Ride along as this company evolves from constantly shipping directly to production to a robust staging and internal canary deployment system.

  Greg Foster — Graphite

A lighthearted but still detail-filled take on a post-incident analysis for a short production outage.

  Greg Foster — Graphite

This one has an interesting discussion of the nature of reliability and the impact of multiple services on overall reliability, including possible mathematical models to use.

  Fitz — Temporal

This episode of the SREPath Podcast covers a variety of themes around observability and SLOs. There’s a great text-based summary if that’s your preference.

  Ash Patel — SREPath

This piece argues that you should install system debugging tools in on your production systems now, because it’s going to be really hard to do it live when you need them.

  Brendan Gregg

Following on from a previous article about the squiggliness of availability numbers, this article evaluates SLAs from 4 major companies to try to divine what they actually mean.

  Ross Brodbeck

I want to present real-life examples of both availability and latency SLOs, as they are more nuanced than they may initially appear. Also, I find it worthwhile sharing a detailed guide as it showcases uncommon uses of PromQL and demonstrates the language’s versatility.

  Michał Kaźmierczak

SRE Weekly Issue #418

A message from our sponsor, FireHydrant:

FireHydrant is now AI-powered for faster, smarter incidents! Power up your incidents with auto-generated real-time summaries, retrospectives, and status page updates.
https://firehydrant.com/blog/ai-for-incident-management-is-here/

The observability waters have been muddy for awhile, and this article does a great job of taking a step back and building a definition — and a roadmap.

  Hazel Weakly

Fred Hebert wrote this response/follow-on to Hazel’s article:

The main points I’ll try to bring here are on the topics of the difference between insights and questions, the difference between observability and data availability, reinforcing a socio-technical definition, the mess of complex systems and mapping them, and finally, a hot take on the use of models when reasoning about systems.

  Fred Hebert

What the service providers are willing to put on the table in terms of penalties is often much less than the money you lose when your service goes down.

  Alex Ewerlöf

Fascinating legal questions come to the surface when lawyers consider the possibility for legal risk exposure from a surgical incident debriefing meeting.

  Dr. Rob Poston

if you approach on-call the right way, you can mitigate the impacts of alert fatigue or, better yet, avoid it altogether. Here, we’ll dive into the tactics teams can implement to address alert fatigue and its underlying causes.

  incident.io

How do you create an SLO that references multiple SLIs together, such as slow requests and errors?

  Ross Brodbeck

More than just a list of talks, this piece pulls out major themes from SRECon24.

  Will Gallego

Making your 9’s look great by cheating.

Of course, you don’t actually want to do that, but learning how can show us that availability numbers are nuanced.

  Ross Brodbeck

SRE Weekly Issue #417

A message from our sponsor, FireHydrant:

Join FireHydrant this Thursday for a conversation about on-call burnout and how to prevent it. Get a better understanding of what makes a fatigue-free on-call culture, including real-world examples from your incident management peers. No sales, just shop talk.
https://app.livestorm.co/firehydrant/better-incidents-spring-bonfire-secrets-to-fatigue-free-on-call-in-2024

Remember that cool lava lamp random number generator that Cloudflare uses? Now they have a couple of other sources of entropy, and they’re teaming up with other companies.

  Cefan Daniel Rubin, Luke Valenta, and Thibault Meunier — Cloudflare

To support 123 million simultaneous streams (!), Paramount+ migrated to a multi-region architecture with a distributed, multi-write database.

  Denis Magda — Yugabyte

DevOps Research and Assessment or the Digital Operational Resilience Act, which is which? Turns out they both matter to SREs.

  Lee Fredricks — PagerDuty

2038 isn’t so far off now. Do you have a plan for 64-bit timestamps?

  Code Reliant

To ensure they would dogfood the new account process regularly, these folks delete a random employee’s account in their product every day.

  Greg Foster — Graphite

Hey, check it out, sidecars are going to be fully supported in upcoming versions of Kubernetes!

  Steven Aldinger — TeamSnap

As part of releasing a new product, FireHydrant ran simulations to determine the right SLO — and uncover some room for optimization.

  Danielle Leong — FireHydrant

  This article is published by my sponsor, FireHydrant, but their sponsorship did not influence its inclusion in this issue.

If you’re new to distributed tracing, this is a great overview. The part about automated instrumentation for span tracing is especially useful.

  Chris Battarbee Metoro

SRE Weekly Issue #416

A message from our sponsor, FireHydrant:

We need tools that help us show our value, enhance understanding of our systems, and free time for us to expand our skills. In this article, FireHydrant lays out three questions to ask vendors as you evaluate DevOps tools. https://firehydrant.com/blog/3-questions-to-ask-of-any-devops-tool-in-2024/

What can we, in turn, learn from some of the most honest and blameless—and public—postmortems of the last few years?

They cover incidents from GitLab, Tarsnap, Roblox, and Cloudflare with great summaries and takeaways.

  The Hacker News

My favorite part of this interview is when Vanessa describes parenting twin babies as constant incident response.

  Shane Hastie — InfoQ

Here follow some lessons I’ve learned from the trenches in small start-ups and larger engineering teams, to improve your on-call shift experience and remediation time for production issues and make sure you’re spending on-call efforts on what has the most impact.

  Alex Wauters

Doing your chaos experiments in a non-production environment can feel safer, but what are you giving up?

  Sam Rossoff — Gremlin

Sometimes, shell is just the right tool for the job.

  Amin Astaneh — Certo Modo

Catherine from Mastodon summarized this incident report beautifully:

this is one of the most violently unhinged CSB reports i’ve ever read […]

while investigating an explosion at a facility, CSB staff tried to prevent another explosion of the same kind in the same facility, and being unable to convince the workers to not cause it, ended up hiding behind a shipping container

  U.S. Chemical Safety and Hazard Investigation Board

This one’s about why people tend to want a “SPoG” and what we should want instead. Bonus points for the Star Trek reference.

  Nočnica Mellifera — Checkly

Right in the middle of migrating from one datacenter to an HA pair of new datacenters, one of the new ones failed. They had to quickly do a partial rollback of the migration to ride out the outage.

  Gauthier François — Doctolib

Today, we are thrilled to announce the release of bpftop, a command-line tool designed to streamline the performance optimization and monitoring of eBPF programs.

  Jose Fernandez — Netflix

SRE Weekly Issue #415

A message from our sponsor, FireHydrant:

Join FireHydrant and talk shop with your DevOps peers on March 28! You’ll gain a better understanding of what makes a fatigue-free on-call culture and how to implement practices to improve yours at this free, virtual roundtable.
https://app.livestorm.co/firehydrant/better-incidents-spring-bonfire-secrets-to-fatigue-free-on-call-in-2024

[…] it must be said that the intent of these metrics was always to give an indicator of how well your team was delivering software, not a high-stakes metric that should be used, for example, to hire and fire team leads.

  Nočnica Mellifera — The New Stack

A primer on the problems with N+1 database queries and how this pattern can sneak into your code whether you realize it or not.

  neda — ReadySet

A great explainer on choosing the right SLIs, starting with the Golden Signals and branching out.

  Tyler Treat

My favorite part about this is the “latency budget” question — which team’s code gets to spend how much time doing its part to serve a request?

  Alex Ewerlöf

Changes in two programs outside the container made Ceph suddenly grind to a halt, as detailed in this troubleshooting story.

  Vladimir Guryanov — Palark

The word “one” is the key here, as the author argues for getting rid of “warning” alerts entirely in favor of using only “critical”.

  Gauthier François

They wrote a Slack bot to summarize open PagerDuty incidents every day.

  Matt Weingarten

The problems I’ll explore in this blog—from the SRE perspective—are about time pressures (when to ship the investigation) and the type of report people expect.

  Fred Hebert — Honeycomb

  Full disclosure: Honeycomb is my employer.

In order to reduce the noise, first they had to define noisy alerts and the KPIs they were looking to improve.

  Gauthier François — Doctolib

A production of Tinker Tinker Tinker, LLC Frontier Theme