SRE Weekly Issue #23

Articles

Here’s the talk on Heroku’s SRE model that fellow SRE Courtney Eckhardt and I gave at SRECon16 in April. Heroku uses a “Total Ownership” model for service operations, meaning that individual development teams are responsible for running and maintaining the services that they deploy. This in turn allows SRE to broaden our scope of responsibility to cover a wide range of factors that might impact reliability.

Full disclosure: Heroku, my employer, is mentioned.

RushCard to pay $19 million to users for last year’s outage

RushCard is a prepaid debit card system, and last year they had an outage that lasted for two weeks. As part of a settlement, RushCard will pay affected customers $100 – $500 for their troubles.

Many RushCard customers are low-income minority Americans who don’t have traditional bank accounts. Without access to their money stored on their RushCards, some customers told The Associated Press at the time they could not buy food for their children, pay bills, or pay for gas to get to their jobs.

Safety Reporting Leads to Safer Systems

This article in Brigham and Women’s Hospital’s Safety Matters series highlights the importance of encouraging reporting of safety incidents and a blameless culture. Two excellent case studies involving medication errors are examined.

Report ET2016: Fire on board a freight shuttle in the Channel Tunnel

In early 2015, a fire occurred in the Channel Tunnel. Click through for a summary of the recently-released post-incident analysis. It includes the multiple complicating factors that made this into a major incident plus lots of remediations — my favorite kind of report.

How We Monitor and Run Kafka at Scale

SignalFx shares their in-depth experience with Kafka in this article. This reminds me of moving around ElasticSearch indices:

Although Kafka currently can do quota-based rate limiting for producing and consuming, that’s not a applicable to partition movement. Kafka doesn’t have a concept of rate limiting during partition movement. If we try to migrate many partitions, each with a lot of data, it can easily saturate our network. So trying to go as fast as possible can cause migrations to take a very long time and increase the risk of message loss.

Auto-Scaling and Self-Defensive Services in Golang

Plagued by pages requiring tedious maintenance of a Golang process, this developer sought to make the service self-healing.

Step-by-Step High Availability with Docker and Java EE

For the Java crowd, Oracle published this simple guide on writing and deploying highly available Java EE apps using Docker. Sort of. Their example uses a single Nginx container for load balancing.

Outages

Salesforce.com
- One of 37 US Salesforce pods went down for over 18 hours. Data from a five-hour period was lost.
  
  Full disclosure: Salesforce.com (parent company of my employer, Heroku), is mentioned.
Time Warner and Cox (ISPs)
- Level3 fiber cut in New York City.
Montreal emergency responders’ radio communication network
UK Post Office
Betfair US
- Betfair’s site failed during the Kentucky Derby, and the CEO publicly blamed “human error”. This article suggests another theory:
  
  While there’s no proof that TVG came under DDOS attack on Saturday, Betfair US may have decided it was more advantageous for customers to think of them as occasionally ham-fisted stooges prone to disabling their own technology rather than an operator whose systems were unable to defend themselves – and their customers’ data – against malicious third parties.
Second Life
- Central DB failure plus a coincidental status site failure. Thanks to April Linden for the detailed report.
Software Update Destroys $286 Million Japanese Satellite
AppleID site
Intuit
Telstra DNS outage
Loblaw Stores
- All stores in the Loblaw chain in Canada, including Shopper’s Drug Mart, No Frills and Real Canadian Superstore, have been forced to close this morning due to “tech issues.”
Walmart, Lowe’s online stores
Afrihost Mobile
- They’re granting 1GB of free data to customers by way of apology.
Multiple AU Banks’ ATMs

SRE Weekly Issue #23

Articles

Outages

Subscribe

RSS

Mastodon

Search Issues