In this episode of the podcast Page it to the Limit, they discuss learning how to be an incident commander.

There was major AWS outage and the second day I was incident command.

  Kat Gaines, with guest Iris Carrera — Page it to the Limit

This article discusses three aspects of fully owning your systems: mandate, knowledge, and responsibility. After defining those terms, it goes on to discuss what happens if one of the three is missing.

  Alex Ewerlöf

I really like the “Managing High RPS” section, especially the part about ignoring events if they’re too old to be relevant any longer.

  Ankush Gulati and David Gevorkyan — Netflix

Cool idea! When a process is overloaded, the system drops requests based on heuristics until the overload condition has passed.

  Bryan Barkley — LinkedIn

Here’s another take on incident severity and priority levels. The two terms are different and mean specific things.

  Robert Ross — FireHydrant

Can we please agree to stop calling them “postmortems”?

  Ash P — Cruform Newsletter

The term “service level” goes back to the US highway system maintenance procedures, among others.

  Akshay Chugh and Piyush Verma — Last9

Charity Majors has railed against metrics for years. Now, her company Honeycomb has a metrics product offering. How does she square it?

  Charity Majors — Honeycomb

Despite the December AWS outage, folks aren’t fleeing AWS, and multi-cloud designs for reliability still don’t make sense, according to this cloud consultant. The media angle is fascinating.

  Lydia Leong — Cloud Pundit

This article has a great list of ideas of who to talk to, plus a section on how to prioritize when you’re short on time.

  Daniela Hurtado — Jeli

Here’s the next incredibly useful article in Jeli’s Incident Analysis 101 series. This one covers the skills and traits of a good incident analyst, along with what not to look for.

  Laura Maguire — Jeli

This article has a remarkable level of detail on 13 incidents at Twitter that were related to cache. The authors open with an explanation of why they focused on cache-related incidents.

  Dan Luu and Yao Yue

[…] the same three pillars form the core of any good process, whether it’s for the largest e-commerce giant or a scrappy SaaS startup.

The three pillars are:

1. Clarity
2. Transparency
3. Calm

  Lisa Karlin Curtis — incident.io

This one recommends doing away with “P0” and “P5” and instead using plain words like “Low” and “High”.

  Stephen Whitworth — incident.io

Feature flags can be a useful way to resolve user impact during an incident.

  Weihan Li — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Implementing a dead-switch for your alerting tool is really important so that you don’t blissfully sleep through an outage.

  Chris Loukas — HelloFresh

As SRE #1, the author of this article got to define the SRE role from the ground up.

  Fred Hebert — Honeycomb

In this article, I will share five lessons I learned about starting SRE teams (or engagements, or organizations).

This article is all about the shape of an SRE team, rather than technical details like SLOs and such.

  Andrea Spadaccini — USENIX ;login:

Why do we use the term “root cause”? I especially love the opening analogy.

  John Allspaw — The ReadME Project

Our Reliability Manifesto is a succinct collection of rules, guidelines, and best practices that reflect our current thinking on what it takes to build a reliable system.

  Christian Hardenberg — Delivery Hero

Here’s how New Relic sets their S*Os.

Set SLIs and SLOs against system boundaries

  Dan Holloran and Elisa Binette — New Relic

It involves lots of machine learning and a “team resilience score”.

  Jennifer Riggins — The New Stack

Every incident is unique, so incident analysis is about learning in order to improve resilience, rather than trying to “fix” a “root cause”.

  Laura Maguire — Jeli

A lot of incident management guides out there are aimed at established, big-scale companies. Things are different when you’re in startup mode.

  Chris Evans — incident.io

This is so cool! It’s a guide for what kinds of incidents you’re likely to learn the most from. There’s a long list of things to look out for with explanations.

  Laura Maguire and Vanessa Huerta Granda — Jeli

Oh, now this is fascinating. Firefox, like, the web browser itself, had an outage in January. It just stopped working for everyone.

  Christian Holler — Mozilla

If you’re looking for an explainer on the CAP theorem, this article gives a great overview with practical details.

  Bartłomiej Żyliński — SoftwareMill

This is about what the security field can learn from SRE. Obviously not directly applicable to SRE, but this article gives us a really great outside perspective.

  Anton Chuvakin — Security Boulevard

Code doesn’t “rot”, but the environment around it changes constantly.

  Lorin Hochstein

How do complex service dependencies affect your SLA? What if service A depends on service B and C being up, but service D or E being up?

TLDR; for serial, multiply availability; For parallels, multiply unavailability.

  Alex Ewerlöf

Here’s how and why not to be a hero. It’s bad for you and everyone else too.

  Isaac Seymour — incident.io

This followup to their initial incident report has a lot to learn from, especially if you run Consul at scale.

  Daniel Sturman and others — Roblox

This week, I came across the Byford Dolphin diving bell incident. This accident seems at face value to be “human error”, but there’s so much to it. Content warning: the accident was quite grisly.

  Wikipedia

Canary testing is more than just deploying your code to a small part of your fleet. You need a plan for how you’re going to spot problems.

  Jyoti Sahoo — OpsMx

My favorite part is how they look for changes in performance, rather than using a static threshold.

  Angus Croll — Netflix

It pays to think ahead about how you’ll answer questions from execs during an incident.

  Chris Fenning — DZone

On January 24, 2022, as a result of an internal Cloudflare product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin.

  Jeremy Hartman — Cloudflare

An analysis of SRE job descriptions from 4 companies highlights what businesses actually expect SREs to do.

  JP Cheung — Rootly
This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Members of the search giant’s site reliability group say managers fostered a toxic environment. Google says a ‘safe, inclusive workplace’ is a top priority.

  Nico Grant — Bloomberg