General

SRE Weekly Issue #300

300 issues. 6 years. Wow! I couldn’t have done it without all of you wonderful people, writing articles and reading issues. Thanks, you make curating this newsletter fun!

A message from our sponsor, Rootly:

Manage incidents directly from Slack with Rootly πŸš’. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:
https://rootly.com/demo/?utm_source=sreweekly

Articles

This is the best thing to hit incident analysis since the Etsy Debriefing Facilitation Guide and the PagerDuty retrospective guide!  This one’s even better because it’s not just about the retrospective, but the whole incident analysis process.

BONUS CONTENT: A preview/introduction by Lorin Hochstein.

  jeli

SysAdvent is back!

When teams only consult briefly on reliability or operational concerns, often the final output doesn’t adequately reflect customer or engineering expectations of reliability of the product or operability of the internals.

  Martin Smith (edited by Jennifer Davis) β€” SysAdvent

What can Dungeons and Dragons teach us about SRE?

  Jennifer Davis β€” SysAdvent

It’s so true. Don’t forget to read the alt text.

  Randall Munroe

This talk (with transcript) includes three stories about how incident analysis can be super effective.

  Nora Jones β€” InfoQ

I know this is SRE Weekly and not Security Weekly, but this vulnerability is so big that I’m sure many of us triggered your incident response process, and some of us may have even had to take services down temporarily.

  John Graham-Cumming β€” Cloudflare

What a colorful metaphor. This article discusses an effective technique for breaking up a monolith, one piece at a time.

  Alex Yates β€” Octopus Deploy

This article proposes a method of eliminating the need for a central team of architects, and it strikes me as very similar to the practice of SRE itself.

  Andrew Harmel-Law

More from the VOID, this piece is about the importance of analyzing “near miss” events.

  Courtney Nash β€” Verica

If you load-test in production, don’t include your load-test traffic in your SLO calculation.

  Liz Fong-Jones β€” Honeycomb

Outages

  • AWS us-east-1 region (and half the web)
    • Between the AWS outage and log4j, it’s been a busy week. Amazon has already posted a write-up about the incident, which includes the notable tidbit that their circuit-breaker/back-off code failed.

SRE Weekly Issue #299

A message from our sponsor, Rootly:

Manage incidents directly from Slack with Rootly πŸš’. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:
https://rootly.com/?utm_source=sreweekly

Articles

Lacking enough incidents to learn from, NASA “borrowed” incidents from outside of their organization and wrote case studies of their own!

Β Β John Egan β€” InfoQ

In this interview, they hit hard on the importance of setting and adhering to clear work hours when working remotely as an SRE.

Β Β Ben Linders (interviewing James McNeil) β€” InfoQ

Here’s a clever way to put a price on how much an outage cost the company.

Β Β Lorin Hochstein

This article introduces error budgets through an analogy to feedback loops in electrical engineering.

  Sjuul Janssen β€” Cloud Legends

[…] saturation SLOs have always been a point of discussion in the SRE community. Today, we attempt to clarify that.

  Last9

Here’s how the GitHub Actions engineering team uses ChatOps. I love the examples!

  Yaswanth Anantharaju β€” GitHub

This contains some pretty interesting details on their major outage last month.

  GitHub

In the last few weeks, I’ve been working on an extendible general purpose shard coordinator, Shardz. In this article, I will explain the main concepts and the future work.

Lots of deep technical detail here.

  Jaana Dogan

They constructed a set of git commits, one for each environment variable, then used git bisect to figure out which variable was causing the failure. Neat trick!

  Diomidis Spinellis

Outages

SRE Weekly Issue #298

Email subscribers, my apologies for the double-send last week. I upgraded WordPress and subsequently further cemented my distrust of all version upgrades ever.

I carefully tested a fix in staging before rolling it out gradually in preparation for this week’s issue. Just kidding, I hacked on it live until I got it fixed. Sorry about all those testing tweets. #testinproduction #yolo #SREWeeklydoesnotpracticeSRE

A message from our sponsor, Rootly:

Manage incidents directly from Slack with Rootly πŸš’. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:

https://rootly.com/?utm_source=sreweekly

Articles

This is Google’s detailed report from their outage last week. This one’s really worth a read; I promise you won’t be disappointed!

  Google

I really like this guide and template for writing incident reports. Each section comes with an explanation of what goes there with examples.

  Lorin Hochstein

Booking.com developed their Reliability Collaboration Model to guide the engagement between SRE and product development teams and the responsibilities assigned to each.

  Emmanuel Goossaert β€” Booking.com

Especially timely now, in the thick of the holiday on-call period.

  James Frost β€” Ably

Great tips. I hope your Black Friday / Cyber Monday is going well!

  Quentin Rousseau β€” Rootly

This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

I thought it might be better to try a new approach: defining what SRE was by looking at what it’s not. Or to put it another way, what can you remove from SRE and have it still be SRE?

  Niall Murphy

Instead of asking that question this article urges understanding what happened.

Another reason that imagining future scenarios is better that counterfactuals about past scenarios is that our system in the future is different from the one in the past.

  Lorin Hochstein

Outages

SRE Weekly Issue #297

A message from our sponsor, Rootly:

Manage incidents directly from Slack with Rootly πŸš’. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:
https://rootly.com/?utm_source=sreweekly

Articles

It’s that time of year again, but maybe it’s time to rethink that code freeze.

Robert Ross β€” FireHydrant

This article really gets to the heart of why I love a good incident. I mean, obviously, I want to minimize, incidents. I swear.

Lisa Karlin Curtis β€” incident.io

This article draws on incident reports from The VOID to show how root cause analysis can be problematic.

Courtney Nash β€” Verica

It’s interesting to read this article after reading the previous one. In the “my car won’t start”, I found myself immediately wondering, why was the vehicle not maintained? What factors contributed to that?

SΓΈren Pedersen β€” Dzone

These are the “phases”, although they stress that aiming for Visionary doesn’t make sense for all organizations.

  • Absent
  • Reactive
  • Proactive
  • Strategic
  • Visionary

Google

Not the field I would have expected to look to for lessons, but it totally works!

Paul Marsicovetere β€” Formidable

This article introduces a 3-phased approach for safe database schema changes: Expand, Rollout, and Contract.

Alex Yates β€” Octopus Deploy

Try to run a program, and you get “No such file or directory”, even though the program is right there. How can this happen?

Julia Evans

Outages

  • Google Cloud Load Balancing
    • Google had a major outage that took down many sites and services. Notably, users of these sites were greeted with a Google 404 page with no branding related to the site they were attempting to access.
  • Grab
  • Tesla
    • Tesla owners were locked out of their cars or unable to start them during the outage.

SRE Weekly Issue #296

A message from our sponsor, Rootly:

Manage incidents directly from Slack with Rootly πŸš’. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo:
https://rootly.com/?utm_source=sreweekly

Articles

WOW! This is the longest, most detailed public incident post I’ve ever seen from any company. I’ve linked to their short(er) summary, but be sure to check out the long version for all the juicy details.

If we operate too far from the edge, we lose sight of it and can’t anticipate when corrective work should be emphasized. If we operate too close to it, we are constantly in high-stakes situations and firefighting.

Fred Hebert β€” Honeycomb

This article goes through the actual math of creating an alert for an SLO, including how to avoid alerting for the entire sliding window even after the problem is fixed.

Ervin Barta

This reddit thread doesn’t have any firm answers, but the discussion is pretty interesting.

u/faidoc and others β€” reddit

Good advice for writing resumes in general, with some SRE-specific tips. There are also links to example SRE resumes.

Quentin Rousseau β€” Rootly

This article is published by my sponsor, Rootly, but their sponsorship did not influence its inclusion in this issue.

Turns out they have runbooks too β€” or I guess you could say we have SOPs.

Hugh Brien β€” Transposit

What do you do about developers that just don’t want to be on call?

Charity Majors β€” Honeycomb

Before opening their new API up to the public, Ably walloped it with Locust.

Denis Sellu β€” Ably

Outages

SRE WEEKLY © 2015 Frontier Theme