A message from our sponsor, Planetscale:
Most database incidents start with one expensive query, not the database being down. PlanetScale gives SRE teams high-availability Postgres and MySQL with automated failover, query insights, and Database Traffic Control to stop runaway queries before they page you.
What can you do to shorten the time to detect an incident? Some great ideas in here, especially monitoring your company’s main web page for a sudden uptick in traffic.
Brent Chapman
What an interesting incident! I recommend reading Azure’s write-up before reading Lorin’s excellent analysis.
Lorin Hochstein
Distributed databases rarely fail in the clean, isolated ways described by component diagrams. They fail through timing gaps, stale metadata, ambiguous ownership, retry storms, incompatible health decisions, and overlapping maintenance activity.
Varsha Ganesh — DZone
I love this concept of a “political incident”:
The subject was political incidents, by which I mean the ones where the severity arrives before the impact assessment does.
And ouch, I felt this bit:
You have spent forty minutes of the incident on the severity field.
Tim Irving
Where can you safely use LLM agents, versus when you should keep things in human hands? This one has some good criteria to consider.
Sai Joshitha Kathari — HackerNoon
I learned a lot about Git while reading this one. Speeding up Git clones in CI may not seem important, but it will when you’re trying to roll out a fix during an incident.
Mike Thompson and Daniel Esponda — Datadog
Switching from their custom-written autoscaler to the new off-the-shelf option made sense, but it wasn’t a simple drop-in replacement.
Samuel Yeboah, Francesco Di Chiara and Mingliang Liu — Netflix
Can we replace human code review with LLM-based reviews? This article lays out what an LLM can’t replicate, and I’d argue that these are the pieces that matter most for reliability.
John Allspaw — Adaptive Capacity Labs