Should you go multi-cloud? What should you do during an incident involving a third-party dependency? What about after? Read this one for all that and more.

  Lisa Karlin Curtis — incident.io
Full disclosure: Fastly, my employer, is mentioned.

An introduction to the concept of common ground breakdown, using the Uvalde shooting in the US as a case study.

  Lorin Hochstein

The comments section is full of some pretty great advice, including questions you can ask while interviewing to suss out whether the on-call culture is going to be livable.

  u/dicksoutfoeharambe (and others) — reddit

From the archives, this is an analysis of a report on the 2018 major outage at TSB Bank in the UK.

  Jon Stevens-Hall

You can determine whether backoff will actually help your system, and this article does a great job of telling you how.

  Marc Brooker

I’ve read (and written) plenty of IC training guides, but this is the first time I’ve come across the concept of a “Hands-Off Update”. I’m definitely going to use that!

  Dan Slimmon

This is a really great exlpanation of observability from an angle I haven’t seen before.

a metric dashboard only contributes to observability if its reader can interpret the curves they’re seeing within a theory of the system under study.

  Dan Slimmon

They asked four people and got four answers that run the gamut.

  Jeff Martens — Metrist

How Airbnb automates incident management in a world of complex, rapidly evolving ensemble of microservices.

Includes an overview of their ChatOps system that would make for a great blueprint to build your own.

  Vlad Vassiliouk — Airbnb

Rigidly categorizing incidents can cause problems, according to this article.

From the customer’s viewpoint… well why would they care what kind of technical classification it is being forced into?

  Jon Stevens-Hall

Lots of great advice in this one.

• If no human needs to be involved, it’s pure automation.
• If it doesn’t need a response right now, it’s a report.
• If the thing you’re observing isn’t a problem, it’s a dashboard.
• If nothing actually needs to be done, you should delete it.

   Leon Adato — New Relic

Using the recent Atlassian outage as a case study, this article explains the importance of communication during an incident, then goes over best practices.

  Martha Lambert — incident.io

My favorite part about this is the advice to “lower the cost of being wrong”. Important in any case, but especially during incident response.

  Emily Arnott — Blameless

There are some interesting incidents in this issue: one involving DNS and another with an overload involving over-eager retries.

  Jakub Oleksy — GitHub

A great read both for interviewers and interviewees.

  Myra Nizami — Blameless

Their main advice is to avoid starting with a microservice architecture, and only transition to one after your monolith has matured and you have a good reason to do so.

  Tomas Fernandez and Dan Ackerson — semaphore

Their notification service had complex load characteristics that made scaling up a tricky proposition.

  Anand Prakash — Razorpay

Coalescing alerts and adding dependencies in AlertManager were the key to reducing this team’s excessive pager load.

  steveazz — GitLab

Lorin Hochstein has started a series of blog posts on what we can learn about incident response from the Uvalde school shooting tragedy in the US. This article looks at how an organization’s perspective can affect their retrospective incident analysis.

  Lorin Hochstein

My claim here is that we should assume the officer is telling the truth and was acting reasonably if we want to understand how these types of failure modes can happen.

Every retrospective ever:

We must assume that a person can act reasonably and still come to the wrong conclusion in order to make progress.

  Lorin Hochstein

How do you synchronize state between multiple browsers and a backend, and ensure that everyone’s state will eventually converge? These folks explain how they did it, and a bug they found through testing.

  Jakub Mikians — Airspace Intelligence

MTTR is a mean, so it doesn’t tell you anything about the number of incidents, among other potential pitfalls.

  Dan Slimmon

Last week, I included a GCP outage in europe-west2. This week, Google posted this report about what went wrong, and it’s got layers.

Bonus: another GCP outage report

  Google

Meta wants to do away with leap seconds, because they make it especially difficult to create reliable systems.

  Oleg Obleukhov and Ahmad Byagowi — Meta

If you’re anywhere near incident analysis in your organization, you need to read this list.

  Milly Leadley — incident.io

I’ve been listening to this podcast this week and I love it! Each episode covers a disaster, safety theory, and other topics — with no ads. Their site is down right now, but the podcast is available on the usual platforms.

  Drew Rae — DisasterCast

If we want to get folks to own their code in production, we need to teach them how to think like an SRE.

  Boris Cherkasky

Let’s look at three mistakes I’ve made during those stressful moments during the beginning of an incident — and discuss how you can avoid making them.

The mistakes are:

Mistake 1: We didn’t have a plan.
Mistake 2: We weren’t production ready.
Mistake 3: We fell down a cognitive tunnel.

  Robert Ross — FireHydrant

At what point does your canary test indicate failure? Should the criteria be the same as your normal production alerting?

  Øystein Blixhavn

This is a followup to a previous article about on-call health. In this one, the author shares metrics about the number of alerts and discusses whether this number is useful.

  Fred Hebert — Honeycomb

Their dashboard crashed for 50% of user sessions, so they had a lot of work ahead of them. Find out how they got crash-free sessions to 99.9% and improved their time to respond to incidents.

  Sandesh Damkondwar — Razorpay

Rogers Communications, a major telecom in Canada, had a country-wide outage earlier this month. I don’t normally include telecom outages in the Outages section because they rarely share information that we can learn from.

This time, Rogers released a (redacted) report on their outage, and this Twitter thread summarizes the key points.

  @atoonk on Twitter

Is your status page status.yourcompany.com? If so, read this article, then get yourself a new domain.

  Eduardo Messuti — Statuspal

The author used my favorite technique for getting up to speed on a company: analyzing a recent incident.

  Vanessa Huerta Granda — Jeli

There are a number of lessons I learned guiding weeks-long backcountry leadership courses for teens that I carried with me into my roles in incident management. In this blog post, I’ll share three that stand out.

  Ryan McDonald — FireHydrant

I really like these articles about interpreting SRE in a way that makes sense for your organization. SRE is still constantly evolving.

  Steve Smith — Equal Experts

The author led an incident just 3 months into their tenure. Here’s what they learned.

  Milly Leadley — incident.io

while SRE and DevOps type job explainers have been written ad nauseam, I found there’s relatively little online about Observability Teams and roles. I figured I’d share a bit about my experience on an O11y Team.

  Eric Mustin

I found the contrast between this one and the previous article interesting. The previous one includes a quote of Brendan Gregg:

Let me try some observability first. (Means: Let me look at the system without changing it.)

  Jessica Kerr — Honeycomb

In June, we experienced four incidents resulting in significant impact to multiple GitHub.com services. This report also sheds light into an incident that impacted several GitHub.com services in May.

  GitHub

Using the Webb telescope as an example, this article describes the progression of a system toward production operation using a metaphor of 3 days.

  Robert Barron — IBM