I’m on vacation enjoying the sunny beaches in Maine with my family, so I prepared this week’s issue in advance. No outages section, save for one big one I noticed due to direct personal experience. See you all next week!
A message from our sponsor, StackHawk:
StackHawk is now integrated with GitHub Code Scanning! Enginners can run automated dynamic application and API security when they check-in code, with results available directly in GitHub.
https://sthwk.com/GitHub-CodeScanning
Articles
We needed a way to deploy our new service seamlessly, and to roll back that deploy should something go wrong. Ultimately many, many, things did go wrong, and every bit of failure tolerance put into the system proved to be worth its weight in gold because none of this was visible to customers.
Geoffrey Plouviez — Cloudflare
I especially like the idea of tailoring retrospective documents to disparate audiences — you may have more than you realize.
Emily Arnott — Blameless
An analysis of two incidents from the venerable John Allspaw. These are from 2012 back when he was at Etsy, and yet there’s still a ton we can learn now by reading them.
John Allspaw — Etsy
An account of how Gojek responds to production issues, and why the RCA is a critical part of the process.
Sooraj Rajmohan — Gojek
Type carefully… or rather, design resilient systems.
JJ Tang — Rootly
Requiring development teams to fully own their services can lead to siloing and redundancy. Heroku works to ameliorate that by embedding SREs in development teams.
Johnny Boursiquot — Salesforce (presented at QCon)
I’ve shared some articles here suggesting doing away with incident metrics like MTTR entirely. This author says that they are useful, but the numbers must be properly ccontextualized.
Vanessa Huerta Granda — Learning From Incidents
Everything could be fine, or we could failing to report or missing problems altogether — we’re flying blind.
Chris Evans — incident.io
