Here’s four of the lessons I learned that should help you build a successful SRE organization.

1. Focus on Developer Training
2. Focus on the Right Abstractions
3. Focus on Self Service
4. Automate Yourself out of a job

  Sven Hans Knecht

In this blog post, we’ll talk about two incident management structure models — distributed and centralized, including the pros and cons of each, and examples of what each structure looks like in our community.

  Robert Ross — FireHydrant

The Rasmussen model conceptualizes the limits of a system along 3 boundaries: Cost, System Performance, and Human Capacity.

  Nishant Modak — Last9

Wow, this is a really interesting incident. it has all the hallmarks of a nightmare sev1: time pressure, unknown problem, inventing new procedures on the spot, multiple different teams/specialties having to work together, etc.

  Jorg Wenninger — CERN

What do you do when many engineers all need to take the same day off each week for religious reasons?

  TimeWeSp

Toyota recently halted production in their factories due to a problem in their order system, about which they shared some interesting details.

  Toyota

Here’s a guidebook on how to handle being the first SRE at a company.

  Sven Hans Knecht

This article makes a cool analogy between designing systems to operate well under unexpected load and designing socio-technical systems that operate well when the people are surprised by what the system is doing.

  Lorin Hochstein

If you need to create SLAs, this article has some solid advice on how to go about it — and what to avoid.

  incident.io

If Prometheus can’t scrape your service, an alert can get resolved incorrectly — and that can happen exactly when your service is failing!

  Chris Siebenmann

A really nifty three-part exploration of action items in the aftermath of an incidents. Rather than consider cost/benefit, this article series proposes that we think about the likelihood of an action item being completed.

  J. Paul Reed

Yes, as it turns out — and these folks have the receipts (along with some theories as to why).

  Colin Bartlett

The “wow” moment in this article is under the heading, “What can we learn from creative desperation?”

  Eric Dobbs — Learning From Incidents

Before explaining how they set up their on-call, these folks share why they avoided it in the early stages of their startup, and what made them finally take the plunge.

  Dustin Brown — DoltHub

For the good of the profession, the SRE community still needs to coalesce around more consistent job ladders, expectations, and competencies.

  Code Reliant

Honeycomb had their worst incident ever at the end of July, and in their characteristic style, they’ve posted an incredibly detailed analysis of what happened — and that’s just the blog post. Then you can click through for a 17-page PDF with lots more detail.

  Fred Hebert — Honeycomb
  Full disclosure: Honeycomb is my employer.

In this post, we’ll explore 10 areas that are key to designing highly scalable architectures.

The 10 areas they cover in-depth are:

1. Horizontal vs. Vertical Scaling
2. Load Balancing
3. Database Scaling
4. Asynchronous Processing
5. Stateless Systems
6. Caching
7. Network Bandwidth Optimization
   8, Progressive Enhancement
8. Graceful Degradation
9. Code Scalability

  Code Reliant

Are you looking at the number of requests that were served successfully out of the total number of requests? Or the percentage of time the system was up and working properly?

  Alex Ewerlöf

This is my personal take on something that is considered standard that I just don’t understand. So here we go — the Apdex, what it is, and why I don’t use it!

  Boris Cherkasky

Here’s a great explanation of three common cognitive biases we should try to avoid while analyzing incidents.

  Randy Horwitz — Learning From Incidents

A horrifying tale of gitops gone wrong and backups that didn’t back up, leading to catastrophic data loss. This, this is what hugops is for. I’m so sorry, Lily!

  Lily Cohen

Here’s a followup analysis from Duo for an incident they had last week.

The first SRE hire at incident.io shares what they learned as they became familiar with the infrastructure and figured out what to do with it.

  Ben Wheatley — The New Stack

This is a story of building a new on-call rotation in a company that didn’t have one. They started out with a pretty awesome list of principles that we could all aspire to.

  Felix Lopez — The New Stack

Why should we test in production? This article gives a really spot-on argument and goes on to explain how to do it.

  Sven Hans Knecht

If “you build it, you run it” requires mandate, knowledge, and responsibility, what happens when one of those is missing?

  Alex Ewerlöf

Slack developed an all-encompassing metric for the user experience that goes beyond a simple SLO.

  Matthew McKeen and Ryan Katkov

This whitepaper delves deep into the ways a microservice architecture changes how transactions work. It presents a method of dealing with microservice transaction failures through application-specific compensation logic.

  Frank Leymann — WSO2

Bambu is a brand of 3d printers that are primarily cloud-based. A problem in their cloud system resulted in printers running jobs unexpectedly, causing significant damage to some customer’s printers.

  Bambu Lab

An interesting confluence of fiber optic line failures resulted in loss of connectivity on what should have been a redundant link.

  Google

I know the title looks like click-bait, but this article delivers with 7 well thought-out critiques of SLOs.

  Code Reliant

This latest entry into the awesome-* arena is a curated list of runbooks and related resources for popular software.

  Runbear

You shift from asking “what was the abnormal work?” to “how did this incident happen even though everyone was doing normal work?”

This article immediately made me think of the latest Mentour Pilot accident investigation in which everyone acted nearly perfectly and yet still only narrowly avoided a mid-air collision.

  Lorin Hochstein

This article will really come in handy next time you need to explain SRE to your execs.

   Kit Merker — DevOps.com

By mapping the Westrum Model of organizational cultures to SRE, we can understand SRE culture adoption.

  Vladyslav Ukis and Ben Linders — InfoQ

Disney’s SRE teams have ensured that the magic keeps happening, even as experiences and their underlying technology become more and more complex.

  Ash Patel — SREPath

There’s so much to learn from this tragedy, I might read this one again. A mid-air collision these days should be effectively impossible due to TCAS. In this case, many factors conspired to bring about disaster.

  Admiral Cloudberg

Here they are, out in the open:

• SLOs create a common understanding in the organization about reliability
• SLOs require investment into improved observability
• SLOs prompt decisions about risk management… and risk-taking

  Amin Astaneh — Certo Modo

The “five standard models” are actually more like a 5-stage workflow:

• Triage,
• Examine,
• Diagnose,
• Test, and
• Cure.

  Saheed Oladosu

This blog post will share broadly-applicable techniques (beyond GraphQL) we used to perform this migration. The three strategies we will discuss today are AB Testing, Replay Testing, and Sticky Canaries.

  Jennifer Shin, Tejas Shikhare, Will Emmanuel — Netflix

Building from a review of traditional rate limiting techniques, this article then explains adaptive rate limiting and its benefits.

  Sudhanshu Prajapati — FluxNinja