A classic from John Allspaw. Designing a resilient system isn’t about eliminating individual causes of downtime; it’s about continuing to operate in spite of them. Allspaw is a big proponent of looking beyond human error to the system surrounding the error.

…human error as a root cause isn’t where you should end, it’s where you should start your investigation.

This could just as well be titled, 7 Rules for Performing Effective Retrospectives. There’s some really great stuff in here and also some good references.

The rules are:

• Learn, don’t blame
• Know the scope of the system
• Make sure you have all the relevant logs
• Make sure the logs lineup with the timeline
• Separate the noise from the information
• Make sure the biases are known
• Make sure you deal in facts and not counterfacts

Spotify shares this deeply technical look at their event delivery and processing system that handles 700k messages per second. The bulk of the article details how they tested Google’s Cloud Pub/Sub to be sure it was reliable enough for their needs.

…Cloud Pub/Sub was being advertised as beta software; we were unaware of any organisation other than Google who were using it at our scale.

Taobao.com suffered a huge security breach in which credentials harvested from previous break-ins were used to break into accounts. This short write-up on DZone urges us to use anomaly detection to catch brute-force attacks like this as they happen.

Reports say the hackers executed approximately 100 million login attempts, and almost 21 million of these turned out to be successful.

Speaking of anomaly detection, this article highlights the problems with existing anomaly detection systems and describes what a successful system would look like. I’ve yet to see a generalized anomaly detection system with an acceptable false positive rate that did better than specific, targeted monitoring.

This survey, released last month, looks possibly interesting. I’m not 100% sure though, because their server is offline and I can’t retrieve it. Oh, the irony.

A deep-dive on EVCache, Netflix’s open source sharding and replication layer on top of memcached.

EVCache is one of the critical components of Netflix’s distributed architecture, providing globally replicated data at RAM speed so any member can be served from anywhere.

This is a guest post from one of our customers, Aaron, Director of Support Systems at CageData. He’s talking about making alerts actionable and why that’s important.

TechCrunch gives us this overview of the field of SRE, including its origins, motivations, and guesses about its future.

Everyone’s favorite OpenSSL vulnerability of the year. I hope you all had a relatively easy patch day.

A short but sweet analysis of an intermittent bug caused by inconsistent date formatting. The author uses the term “blameful postmortem” to mean finding reasons that explain how the client application was written with faulty date parsing logic (tl;dr: the server side truncated trailing zeroes in the fractional seconds). Really, I think this is less about blame than it is about understanding the full context in which a error was able to occur, and that’s exactly what a blameless postmortem is all about.

Incidents can uncover technical debt in a system. Fixing the technical debt is often necessary if a repeat incident is to be avoided, but it can be difficult to track and allocate resources to make it happen. This article from PagerDuty suggests a method for tracking technical debt uncovered by incidents.

When multiple incidents occur simultaneously, things can get hairy and you need to have an organized incident response structure. This article is about firefighting, but we can take their lessons and apply them to SRE.

PagerDuty advocates for a model I’ve heard referred to as “Total Service Ownership”, where dev teams handle incident response for their subsystems rather than “throwing them over the wall” for Ops to support. Courtney and I will be talking about this and more at SRECon16 next month.

What an excellent resource! This repo contains a pile of postmortems for our reading and learning pleasure. I’m linking to the repo now, but I don’t promise not to call out specific awesome postmortems from it in the future.

When you’re in the trenches trying to get the service back up and running, it can be hard to find the time to tell everyone else in your company what’s going on. It’s critically important though, add Statuspage.io writes in this article.

Full disclosure: Heroku, my employer, is mentioned.

Digital Ocean shares this overview of the basic concepts involved in high availability.

This article discusses a method of computing the availability of an overall system made up of individual components with differing availabilities. It gives general formulas and methods that are fairly simple, yet powerful.

What do you do when you have to modify an existing production system that has less-than-wonderful code quality? This article is an impassioned plea to test the heck out of your changes and always try to release production-quality code the first time.

Google is launching a reverse-proxy for DDoS mitigation. Interestingly, it’s only for news and free speech sites and it’s completely free.

The big scary news this week was the buffer overflow vulnerability in glibc’s getaddrinfo function. Aside from the obvious impact of intrusions on availability, this bug requires us to roll virtually every production service in our fleets. If we don’t have good procedures in place, now is when we find out with downtime.

Bryan Cantrill, CTO of Joyent, really crystallized the gross feelings that have been rolling around in my mind with regard to unikernels. I would point colleagues to this article if they suggested we should deploy unikernel applications. He makes lots of good points, especially this one:

Unikernels are entirely undebuggable. There are no processes, so of course there is no ps, no htop, no strace — but there is also no netstat, no tcpdump, no ping!

I find the implicit denial of debugging production systems to be galling, and symptomatic of a deeper malaise among unikernel proponents: total lack of operational empathy.

Atlassian dissects their response to a recent outage and in the process shares a lot of excellent detail on their incident response and SRE process. I love that they’re using the Incident Commander system (though under a different name). This could have (and probably has) come out of my mouth:

The primary goal of the incident team is to restore service. This is not the same as fixing the problem completely – remember that this is a race against the clock and we want to focus first and foremost on restoring customer experience and mitigating the problem. A quick and dirty workaround is often good enough for now – the emphasis is on “now”!

My heart goes out to those passengers hurt and killed and to their families, but also to the controller that made the error. There’s a lot to investigate here about how a single human was in such a position that a single error could caus such devastation. Hopefully there are ways in which the system can be remediated to prevent such catastrophes in the future.

Like medicine, we can learn a lot about how to prevent and deal with errors from the hard lessons learned in aviation.

You’d think technically advanced aircraft would be safer with all that information and fancy displays. Why they’re not has a lot to do with how our brains work.

When I saw Telstra offer a day of free data to its customers to make up for last week’s outage, I cringed. I’m impressed that they survived last Sunday as Australia used 1.8 petabtytes of data.

In this article, the author describes discovering that a service he previously ignored and assumed saw very little traffic actually served a million requests per day.

If ignorance isn’t an operational strategy, what is? Paranoia. You should code and run your systems like a large group of Internet lunatics are out to abuse the hell out of them.

This is a great intro to Chaos Engineering, which is the field I didn’t know existed that was born out of Netflix’s Chaos Monkey. This is the first article in what the author promises will be a biweekly series.

Thanks to Devops Weekly for this one.

So much about what modern medicine has learned about system failures applies directly to SRE, usually without any adaptation required. In this edition of The Morning Paper, Adrian Colyer gives us his take on an excellent short paper by an MD. My favorite quotes:

Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved.

When new technologies are used to eliminate well understood system failures or to gain high precision performance they often introduce new pathways to large scale, catastrophic failures.

The Software Evolution & Architecture Lab of the University of Zurich is doing a study on modern continuous deployment practices. I’m really interested to see the results, especially where CD meets reliability, so if you have a moment, please hop on over and add your answers. Thanks to Gerald Schermann at UZH for reaching out to me for this.

I’ve been debating with myself whether or not to link to Emerson Network Power’s survey of datacenter outage costs and causes. The report itself is mostly just uninteresting numbers and it’s behind a signup-wall. However, this article is a good summary of the report and links in other interesting stats.

Facebook algorithmically generated hundreds of millions of custom-tailored video montages for its birthday celebration. How they did it without dedicating specific hardware to the task and without impacting production is a pretty interesting read.

Administering ElasticSearch can be just as complicated and demanding as MySQL. This article has an interesting description of SignalFX’s method for resharding without downtime.

This is a pretty interesting report that I’d never heard of before. It’s long (60 pages), but worth the read for a few choice tidbits. For example, I’ve seen this over and over in my career:

Yet, delayed migrations jeopardize business productivity and effectiveness, as companies experience poor system performance or postpone replacement of hardware past its shelf life.

Also, I was surprised that even now, over 70% of respondents said they still use “Tape Backup / Off-site Storage”. I wonder if people are lumping S3 into that.

Never miss an ack or you’ll be in even worse trouble.

More on last week’s outage. I have to figure “voltage regular” means power supply. Everyone hates simultaneous failure.

A full seven years after they started migration, Netflix announced this week that their streaming service is now entirely run out of AWS. That may seem like a long time until you realize that Netflix took a comprehensive approach to the migration:

Arguably, the easiest way to move to the cloud is to forklift all of the systems, unchanged, out of the data center and drop them in AWS. But in doing so, you end up moving all the problems and limitations of the data center along with it. Instead, we chose the cloud-native approach, rebuilding virtually all of our technology and fundamentally changing the way we operate the company.