“You can OOM a single NUMA node” thus entered my list of things to worry about when a box seems to have plenty of memory but still goes off and slaughters innocent (but big) processes.

Rachel Kroll

In this podcast episode, the panelists hold a retrospective for the snow-related delay of DevOps Days Baltimore. Toward the end they go into the idea of reliability and single points of failure with respect to conference planning. My favorite quote in the show, from Nell Shamrell-Harrington:

Incident Management is never about technology — it’s a people.

Nell Shamrell-Harrington and Nathen Harvey

I really love this Who, Me? section from The Register.

Simon Sharwood — The Register

This article has a great discussion of how to get started with chaos engineering — and how to avoid biting off more than you can chew.

Jennifer Riggins — The New Stack

Beamer is a stateless datacenter load balancer supporting both TCP and Multipath TCP (MPTCP). It manages to keep the load balancers stateless by taking advantage of connection state already held by servers.

Super-clever! The LB does keep state, but the size of the state is constant, unrelated to the number of connections flowing through it.

Adrian Colyer — summary, Olteanu et al. — original paper

Sometimes it’s worthwhile to lay everything out and describe just exactly what we’re up against as SREs. The analogies here are pretty awesome. Read this for a hefty dose of cynicism about the state of our increasingly computer-driven world.

Peter Welch

If you missed the STELLA Report, released last fall during Velocity NYC by John Allspaw, Richard Cook, and David Woods, this podcast is a great intro. And even if you did catch it, it’s well worth a listen. The Food Fight folks interview John Allspaw and there’s some really stellar (see what I did there) back-and-forth discussion.

Alan Kraft and Nathen Harvey

Great idea. This reminds me of a couple jobs back where I rigged up our infrastructure to log every command entered at the shell into a Slack channel.

Rachel Kroll

This excerpt from the Google SRE book is worth reading if only for this nifty idea for graceful degradation:

Other techniques include […] choosing a consistent subset of clients to receive errors, preserving a good user experience for the remainder.

In part two of this story, the author causes their first incident (oops) and subsequently significantly improves the performance of the system in question (cool!).

Evan Smith — Hosted Graphite

An introduction to blue/green deployments including the risks it helps to alleviate.

Mark Henke — Rollout.io

instead of giving guidelines on how and when to do things, I am going to lay out a few ideas on how to respond to alerts and leave it up to you to decide what methods work best for your app and your organization.

Peter Christian Fraedrich — Capital One

Especially in Ubuntu, it’s harder than it used to be to get a core dump, thanks to apport and the like.

Julia Evans

NCDEX, a stock exchange in Mumbai, India, has been operating out of its disaster recovery site for two weeks. Unfortunately, it looks like performance is not on par with the standard site.

Rajesh Bhayani — Business Standard

You may have heard that a Southwest flight suffered a catastrophic engine failure that left one passenger dead. The day after my family flew a Southwest flight to Orlando. Yikes.

The air traffic control audio recording is incredible to listen to. The pilot that was on the radio was cool and calm as she responded to the incident and arranged for landing and emergency ground crews.

I have different thoughts than the author on a few of the points, but it’s very useful and enlightening to see their thought process.

Will Gallego

What it says on the tin. Pretty neat CI setup!

Bridget Lane — USA Today

Full disclosure: Fastly, my employer, is mentioned.

“Why-run” mode is Chef’s “do nothing” or “dry run” mode. As it turns out, it may not be so useful when trying to figure out what Chef will do.

Julian Dunn — Chef

Lots of deep thoughts on what makes on-call hard and what we can do about it.

Cody Wilbourn

One little typo is all it took.

Rachel Kroll

Q&A about a task queuing system that freezes up if the queue is kept full at all times.

A new hire tells us what it’s like to get up to speed as an SRE at Hosted Graphite.

Evan Smith — Hosted Graphite

Brilliant, just brilliant. This isn’t just another “there isn’t just one root cause” article to skip over. The author takes time to explain the concept with cogent examples and useful metaphors. This one really caught my eye:

What’s the root cause of success?
[…] When building a successful project, there’s never just one thing that goes right for it to succeed.

Will Gallego

This episode of Food Fight is an hour-long interview with guests Rob Schnepp, Ron Vidal, and Chris Hawley, the 3 firefighters behind Blackrock 3 Partners. It’s a great intro to the Incident Management System, and well worth a listen.

Shout-out to Maple Player, an android audio player with a really high-quality tempo increase feature. I was able to listen at 1.5x speed and still understand everything; otherwise, I wouldn’t have had time this week.

Nell Shamrell-Harrington and Nathen Harvey

Here’s one from the archives, an incident report from 2013. After a temporary network partition in a redis cluster, the replicas all tried to resynchronize at once, overloading the master. One of the results was that some customers got repeatedly charged for the same thing.

Twilio

You have to design a system such that the natural thing to do yields a good result and doesn’t put anyone in harm’s way.

Rachel Kroll

I thought consistent hashing was largely solved. I was wrong! There are some good solutions out there, but you have to evaluate their relative trade-offs and pick the right one for your use case.

Damian Gryski

Full disclosure: Damian Gryski is my coworker at Fastly.

As you read this article, consider the ethical imperative of system reliability, when system reliability can literally mean life and death in some cases. That’s only going to be more common in the coming years.

Yonatan Zunger

Our service needs to be available 24/7, without question. In order to ensure this happens, the LogicMonitor TechOps team uses HashiCorp Packer, Terraform, and Consul to dynamically build infrastructure for disaster recovery (DR) in a reliable and sustainable way.

Randall Thomson — LogicMonitor

On Tuesday, 13 March 2018 at 12:04 UTC a database query was accidentally run against our production database which truncated all tables.

Oof. Sorry, Travis folks, but a sincere thanks for sharing your experience with us.

Konstantin Haase — Travis CI

I like these “preliminary results” better than the kinds of aggregate statistics you normally get from a survey report. There are real quotes from free-form survey answers, including a couple of real gems. There’s a link to download the actual survey report if you’re into that, too.

Dawn Parzych — Catchpoint

The BBC suffered two simultaneous major outages that broke their online streaming product and forced their website into a limited-functioning mode.  This post-incident followup explains what happened and how they dealt with it.

Richard Cooper — BBC

Bursting is a hidden reliability risk that has bitten me hard in the past. Click through for an explanation of the risk and how to mitigate it.

Michael Wittig — Cloudonaut

This post has the most concise definition I’ve seen yet for observability, along with a quiz that will tell you whether you’re Doing It Right^TM.

the power to ask new questions of your system, without having to ship new code or gather new data in order to ask those new questions

Charity Majors — Honeycomb

This debugging story is an entertaining read, and it’s also got some useful stuff to watch out for in your systems.

Tick tick tick. Time is hard.

Rachel Kroll

Solid knowledge of how DNS works is critical for SREs. This repo contains an introduction to DNS written to be far more approachable than the (many!) DNS RFCs. It’s a work in progress but there’s a lot of good content already.

Bert Hubert and others

Within this post, we’ll discuss growth planning, the challenges associated with being part of a remote team, and some of the unexpected advantages geographically distributed SRE teams can offer.

Akhil Ahuja — LinkedIn

Her thread starts here and continues being awesome:

Real talk, you should never have a paging alert on a system stats metric. Or a single host anything metric. (Or an aggregate host metric, or an aggregate divided by host count, or …)

Charity Majors