SRE Weekly Issue #301

A message from our sponsor, Rootly:

Manage incidents directly from Slack with Rootly 🚒. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:

https://rootly.com/demo/?utm_source=sreweekly

Articles

This one perhaps belongs in a security newsletter, but the failure mode is just so fascinating. A CDN bug led to the loss of millions of dollars worth of Bitcoin.

  Badger

Google posted a report for the Google Calendar outage last week.

  Google

Jeli, authors of the Howie post-incident guide has their own “howie”. It’s a great example of a thorough incident report.

  Vanessa Huerta Granda — Jeli

Hopefully not too late, here are some tips as we head into the thick of it.

  JJ Tang — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Using their own incident retrospective template, Blameless shows us how to write an incident retrospective.

  Emily Arnott — Blameless

Meta has their own in-house-built tool for tracking and reporting on SLIs.

  A Posten, Dávid Bartók, Filip Klepo, and Vatika Harlalka — Meta

These folks put everyone on-call by default, and also pay them extra automatically for each shift and even covering for coworkers.

  Chris Evans — incident.io

Code that was deployed under a feature flag inadvertently affected all traffic, even with the flag disabled.

  Steve Lewis — Honeycomb

By creating SLOs for microservices at various levels of the request tree, they ended up with a morass of arbitrary targets that didn’t relate clearly to the user experience.

  Ben Sigelman — Lightstep

Outages

  • AWS us-west-1 and us-west-2
    • Hot on the heels of last week’s us-east-1 outage, AWS had a shorter outage in us-west-1 and us-west-2.

  • PagerDuty
    • PagerDuty alert notifications were affected by the AWS us-west-2 outage, and the impact lasted about twice as long as AWS’s.

  • Slack
  • Cloudflare
  • Solana
Updated: December 19, 2021 — 9:30 pm
A production of Tinker Tinker Tinker, LLC Frontier Theme