A message from our sponsor, Buildkite:
More places to run, more scale to manage and maintain, usually means more blind spots; not here. Buildkite’s control plane holds the live state of every job, agent and queue, regardless of throughput size.
See what’s running, what’s waiting and why with immediate insight → https://buildkite.com/platform/pipelines/
People hold post-incident reviews for three separate purposes. When the people that care about each one collide, things can go off the rails.
Brent Chapman
In December, Honeycomb had a major incident, and they posted a pretty detailed write-up on their status page. That was just an interim report though, and this post goes into a ton more detail.
Fred Hebert — Honeycomb
They had a weird problem, and they only really got to the bottom of it when they zoomed out and looked at the effects at the fleet level.
Nathan Bronson — OpenAI
Is the human reviewer able to live up to the assurance they’re supposed to provide?
They are being asked to catch an error at the one moment they have the least context to catch it.
Dan Leiva — CEOWORLD Magazine
The problem is, as an industry we more often than not mistake capturing and archiving information for developing meaningful insights.
Will Gallego — Resilience in Software Foundation
AI lowers the barrier to entry. True. But it also lowers the barrier to overcommitment.
The question isn’t “Can AI help us build this faster?” The question is: “Should we own the infrastructure required to keep this alive for the next five years?”
Bru Woodring — Prismatic
Cross-team latency problems are accountability problems, not just profiling problems. An SLO contract is one way to solve this.
Ujjwal Gulecha — DZone
A guide for building an incident management process at a small company, with a focus on what not to include from the start.
Tim Irving