SRE Weekly Issue #372

A message from our sponsor, Rootly:

Rootly is hiring for a Sr. Developer Relations Advocate to continue helping more world-class companies like Figma, NVIDIA, Squarespace, accelerate their incident management journey. Looking for previous on-call engineers with a passion for making the world a more reliable place. Learn more:


At Pulumi we read every single error message that our API produces. This is the primary mechanism that led to a 17x YoY reduction in our error rate

  Evan Boyle — Pulumi

Rather than striving for a million nines, we should choose the right reliability target based on an evaluation of the effect of downtime on the business.

  Itzy Sabo — HEY

This is a presentation of a study of harm and trauma resulting from incident response work. I especially like the part about blamelessness in theory versus practice.

  Jessica DeVita — InfoQ

Perhaps a sensationalist title, but there’s a really good point here: learning from incidents is only practical if it actually improves the business.

  Chris Evans —

A highly-detailed proposal for a system to track which users are online at a huge scale.

  Nk — System Design

However, for any cache to be used for the purpose of upscaling, it must operate completely independent from the source of truth (SOT) and must not be allowed to fall back to the SOT on failures.

  Estella Pham and Guanlin Lu – LinkedIn

If you design your system to make lying the only viable option, then people will lie. To me, this article is all about understanding that our systems involve real, squishy humans, an designing appropriately.

  Admiral Cloudberg

Updated: May 14, 2023 — 10:45 pm
A production of Tinker Tinker Tinker, LLC Frontier Theme