SRE Weekly Issue #91

Here’s how Hosted Graphite made their job ad for an SRE-like role (Ops Automation Engineer) more inclusive. The article is filled with specific before/after language snippets, each with a detailed explanation of why they made the change.

A couple weeks after their major outage last October, Dyn published this article explaining secondary DNS. It’s a great primer and digs into what to do if you use advanced non-standard functionality like ALIAS records and traffic balancing.

SignalFx goes into deep detail on their feature for predicting future metric values. We get an explanation of why prediction is difficult and a discussion of the math involved in their solution.

Payments: we really have to get them right. Here’s DropBox’s Jessica Fisher with a discussion of how they reconcile failed payments.

No matter what goes wrong, our top priority is to make sure that customers receive service for which they’ve been charged, and aren’t charged for service they haven’t received.

A couple of weeks ago, I linked to a story about Resilience4j, a fault tolerance library for Java. This week is the second installment that shows you how to use it to implement circuit breakers. There’s also an interesting discussion of one of the implementation details.

Here’s a cute little debugging story. Turns out ntpd has a bit of a blind spot!

Adcash CTO Arnaud Granal gives us a rare glimpse into the multiple iterations of their infrastructure. Hear what worked well and what didn’t as they scaled to handle 500k requests per second at peak.