SRE Weekly Issue #2

I’m still working out all of the kinks for SRE Weekly, so the issue for this “week” is hot on the heels of the last one as I clear out my backlog of articles. Coming soonish: decent CSS.

Articles

Managing the burden of on-call is critical to any organization’s incident response. Tired incident responders make mistakes, miss pages, and don’t perform as effectively. In SRE, we can’t afford to ignore this stuff. Thanks to VictorOps for doing the legwork on this!
A talk at QCon from LinkedIn about how they spread out to multiple datacenters.
A review of designing a disaster recovery solution, and where virtualization fits in the picture.
Not strictly directly related to reliability (unless you’re providing ELK as a service, of course), but I’ve found ELK to be very valuable in detecting and investigating incidents. Scaling ELK well can be an art, and in this article, Etsy describes how they set theirs up.
This series of articles is actually the first time I’d seen mention of DRaaS. I’m not sure I’m convinced that it makes sense to hire an outside firm to handle your DR, but it’s an interesting concept.

Outages

A weekend outage for Rockstar.
A large hospital network in the US went down, making health records unavailable.
Snapchat suffered an extended outage.
Anonymous is suspected to be involved.
A case-sensitivity bug took down Snapchat, among other users of Google Cloud.
Google’s postmortem analyses are excellent, in my opinion. We can learn a lot from the issues they encounter, given such a thorough explanation.

SRE Weekly Issue #1

Articles

An excellent discussion of the need to look at human error in a broader context.
A cogent argument that code freezes increase risk rather than reducing it.
An interesting outline of a hardware platform with duplicate everything (cpu, RAM, etc) claiming 7+ nines of availability. I’m not sure I’m convinced of its utility in all but a few niche areas, but it’s a neat concept.
A short discussion of how Netflix prepares for the holidays.
A new tool that checks connectivity though real requests. Is it enough to monitor internal services from a central monitoring machine? What if service A is unreachable only to hosts in cluster B, but Nagios can see it just fine? I’ve seen that before and it made me wonder if I need to monitor everything from everywhere.
Instagram posted an account of growing to double the traffic and multiple regions.

An example of a retrospective analysis process. I especially like this quote:

We also ask what led a person to believe that what they did was the right choice. Rarely does someone intend to do the wrong thing.

What happens when an operator that knows all of the secrets is suddenly unavailable? How do you make their secrets available without compromising security?

Outages

Black Friday, Cyber Monday, and the weekend in between is a critical time for sites to remain available. This year, some notable companies had a hard time.

An older but nice postmortem analysis posted by Slack in October.
PSN was down over black Friday weekend.
Nieman Marcus lost out on much of Black Friday.
Newegg also had issues on Black Friday.
Argos’s Black Friday Deals page was down.
EBay suffered an outage the day before thanksgiving.
PayPal suffered a major outage during cyber weekend.
Google Compute Engine saw issues with some if its transit traffic stemming from a new BGP peer accidentally announcing far more routes than they should have. They posted a nicely detailed analysis.
Target had downtime on Cyber Monday.
Time Warner Cable customers were frustrated by “intermittent availability” (read: outages) on Cyber Monday, hampering their ability to get in on all the deals.
A production of Tinker Tinker Tinker, LLC Frontier Theme