SRE Weekly Issue #89

SPONSOR MESSAGE

Acknowledge and resolve IT & DevOps alerts directly from Slack with the new native integration with VictorOps. Learn all about it here:
http://try.victorops.com/slack/SREWeekly

Articles

Cachet looks like a pretty good contender to incumbents like StatusPage.

Hosted Graphite used PySyncObj to create a fault-tolerant threshold alerting feature.

Talk about a high-pressure incident! When a teleconferencing provider’s wires got crossed, hilarity (and embarassment) ensued.

This article is published by my sponsor, VictorOps, but their sponsorship did not influence its inclusion in this issue.

This story is from a PagerDuty engineer. What’d you learn while shadowing on-call? I’d love to hear your story!

Here’s how SYNQ set their status page up. They’re the folks that committed to publishing all of their incident followups publicly a month or two back. Transparency FTW!

I’ll save you the math: that’s ~17k req/sec. I really like that this article takes us through their learning process and their first failed attempts.

Quid wrote up this explanation of how they set up their game day and what they learned. I really like the structure they used, and I may draw heavily on it for my own game days.

“Observability” as a term is making the rounds like “DevOps” did (and still does…). Here’s Baron Schwartz’s take on it.

Outages

  • Google Services
    • As two astute readers pointed out (thanks!), the Gmail outage I included in the last issue was from 2009(!). Oops. However, Google has been experiencing a series of outages and degradations this month, so I’m just going to pretend I knew that rather than that I forgot to check the date on the article.
  • s3 outage
    • S3 had an outage in us-east-1 on September 14th. This one showed up as yellow on their status site, with the text below. Companies that depend on S3 probably saw impact as well, but I couldn’t find any status posts other than Heroku’s.

      11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
      12:20 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.
      12:38 PM PDT We continue to work towards resolving the increased throttling errors for Amazon S3 requests in the US-EAST-1 Region. We have identified the subsystem responsible for the errors, identified root cause and are now working to resolve the issue.
      12:49 PM PDT We are now seeing recovery in the throttle error rates accessing Amazon S3. We have identified the root cause and have taken actions to prevent recurrence.
      1:05 PM PDT Between 11:40 AM and 12:56 PM PDT we experienced throttling errors accessing Amazon S3 in the US-EAST-1 Region. The issue is resolved and the service is operating normally.

      Full disclosure: Heroku is my employer.

  • IBM
    • IBM had a mishap when transferring control of some of its domains to a different registrar. Some of their services including their Global Load Balancer went down.

SRE Weekly Issue #88

SPONSOR MESSAGE

Acknowledge and resolve IT & DevOps alerts directly from Slack with the new native integration with VictorOps. Learn all about it here:
http://try.victorops.com/slack/SREWeekly

Articles

From Catie McCaffrey:

I’m often asked how to get started with Distributed Systems, so this post documents my path and some of the resources I found most helpful. It is by no means meant to be an exhaustive list.

Julia Evans just blew my mind (once again). In this article, among other things, she links to a tool that tells you which function in the kernel dropped a packet. I’ve been wishing for such a tool for years!

I love that companies are starting to publish lessons learned from game days and other chaos experiments. Just like a post-incident followup, there’s so much we can learn by following along.

It’s an absolute must for any disaster recovery plan worth its name to include power supply as a crucial factor – because, without power, you simply can’t do business.

Here’s the last installment of Jason Hand’s digest version of his new eBook, Post-Incident Reviews.

If I leave you with one take-away from this guide, it should be that every incident provides an opportunity for your team to be more prepared for the next one.

This article is published by my sponsor, VictorOps, but their sponsorship did not influence its inclusion in this issue.

How can you prevent a colo failure? Obviously, colo customers can’t, but we can at least prepare. This article has advice for understanding a provider’s history, policies, and procedures related to outages.

Just click through.

In this analysis of the factors leading to a plane crash, we see another example of the critical role that human/computer interfaces play in allowing (or preventing) humans to recover from a system failure.

Move over, backhoes: water is the other natural enemy of the fiber optic network.

The New York Times has a Kafka installation containing everything they’ve published in their entire history, and it powers the front page, search, suggestions, and everything else.

Outages

  • AbeBooks.com
    • AbeBooks is the place to go for out-of-print books and old editions. The site going down meant that many used booksellers lost a major sales outlet.
  • Gmail
  • Apple developer portal
  • Google Drive
  • iCloud Mail
  • Heroku
    • Heroku posted a pile of public followups this past week:
      • Incidents 1251 and 1254 – In both of these incidents, applications failed due to missing debian packages normally provided by the Heroku platform.
      • Incident 1257 – For a few minutes, 10% of requests to Heroku applications hosted in Europe failed.
      • Incident 1270 – Applications last deployed over 3 years ago spontaneously stopped working.

      Full disclosure: Heroku is my employer.

SRE Weekly Issue #87

SPONSOR MESSAGE

More efficiently reach on-call teams and incident responders with a new way to deploy Live Call Routing using Twilio Functions and VictorOps. Check it out:
http://try.victorops.com/LiveCallRouting/SREWeekly

Articles

John Allspaw describes the Architecture Review Working Group at Etsy. I like the idea of an open discussion with peers before creating a novel system that will add significant operational burden.

Here’s part two of Jason Hand’s series of posts with key takeaways from his new eBook, “Post-Incident Reviews”. In the next three chapters, he shows why a traditional RCA process misses the mark.

[…] problems stem — not from one primary cause — but from the complex interplay of our systems and the teams tasked with managing them.

Honeycomb.io eschews plain monitoring in favor of “observability”, which they define as the ability to “ask any arbitrary question” about a system.

But here’s the thing: in distributed systems, or in any mature, complex application of scale built by good engineers… the majority of your questions trend towards the unknown-unknown.

Here’s another primer on microservices. It has a nice “caveats” section, which is exactly where operations and reliability come into the picture.

Facebook shared a lot of detail about how they evolved from 3 daily pushes to quasi-continuous releases. They’ve got a well-defined canary system, reminding me of Charity’s article on testing in production last week.

AppDynamics presents their list in shiny PDF form. You’ll have to fill in your spam-bucket address contact info to download it.

PagerDuty is hosting a “breakathon”: small teams will compete to resolve a series of infrastructure issues. Sounds like bunch of fun!

Outages

  • Japan
    • Google accidentally announced some BGP prefixes it shouldn’t have, taking Japan offline for a couple of hours. Linked above is a really neat in-depth analysis from BGPmon, for all you BGP geeks out there.

      Since Google essentially leaked a full table towards Verizon, we get to peek into what Google’s peering relationships look like and how their peers traffic engineer towards Google.

  • Heroku
  • AWS
    • EC2’s Ireland region suffered an outage in VPC peering on August 23. Their status site doesn’t allow for deep links, so here’s an excerpt:

      11:32 AM PDT We are investigating network connectivity issues for some instances in the EU-WEST-1 Region.

      11:55 AM PDT We have identified root cause of the network connectivity issues in the EU-WEST-1 Region. Connectivity between peered VPCs is affected by this issue. Connectivity between instances within a VPC or between instances and the Internet or AWS services is not affected. We continue to work towards full recovery.

      12:51 PM PDT Between 10:32 AM and 12:44 PM PDT we experienced connectivity issues when using VPC peering in the EU-WEST-1 Region. Connectivity between instances in the same VPC and from instances to the Internet or AWS services was not affected. The issue has been resolved and the service is operating normally.

  • Google Cloud
    • Google Cloud suffered a massive 30-hour worldwide outage in some cloud load-balancers. In their impressive style, they posted frequent updates during the incident and issued a followup analysis of the incident just 2 days after resolution.

      In order to prevent the issue, Google engineers are working to enhance automated canary testing that simulates live-migration events, detection of load balancing packets loss, and enforce more restrictions on new configuration changes deployment for internal representation changes.

  • WhatsApp
  • Twitch (video streaming service)

SRE Weekly Issue #86

SPONSOR MESSAGE

More efficiently reach on-call teams and incident responders with a new way to deploy Live Call Routing using Twilio Functions and VictorOps. Check it out:
http://try.victorops.com/LiveCallRouting/SREWeekly

Articles

Charity Majors knocks one out of the park with this article on the importance of testing (safely) in production.

Why does testing in production get such a bad rap when we all do it? The key is to do it right.

And speaking of baseball metaphors, here’s a PagerDuty engineer’s first-person account of shadowing on-call during an incident and the lessons she learned.

If you have time, please consider filling out this short survey on post-incident reviews (a.k.a. “retrospectives”) as part of a master’s thesis.

Mathias Lafeldt of Gremlin Inc. gives us this tutorial on moving from hand-run chaos experiments to a fully automated chaos system.

Recently, Jason Hand’s new ebook, Post-Incident Reviews, was published. Here’s his summary of the key points in the first three chapters.

This article is published by my sponsor, VictorOps, but their sponsorship did not influence its inclusion in this issue.

This article describes metrics in three main categories and explains how (and whether) to set up alerts for each kind.

Good output metrics are a close proxy for dollars earned or saved by the system per minute.

Like the previous article, Ilan Rabinovitch of Datadog advocates for symptom-based monitoring and alerting. I like his concept of the improved “durability” of symptom-based alerting (as opposed to cause-based):

[…] you don’t have to update your alert definitions every time your underlying system architectures change.

Our systems are always in flux, and this sometimes leads to failure. Mathias expands on this line of thinking to urge seeking to understand the many conditions that led to a failure, rather than a particular root cause.

Hosted Graphite had a gnarly problem to solve: how to get information about overload conditions from the backend to the front end where throttling could be enacted.

Outages

SRE WEEKLY © 2015 Frontier Theme