ObserveTestIsolateResolve

Public IT incident case studies / Vol. 01

Don’t stop at
the first answer.

Root Cause Casebook is a field guide to how technology teams diagnose public IT incidents. Each evidence-grounded case preserves the hypotheses, wrong turns, decisive tests, and reasoning that narrowed the fault domain.

Study the latest case
Current case

Cloud Infrastructure · Complexity 9/10

Honeycomb EU Kafka outage after DR cleanup triggered leaderless partitions and forced cluster evacuation

After a December 5, 2025 disaster recovery exercise in Honeycomb's EU region, production cleanup steps intentionally destroyed Kafka brokers and left multiple partitions leaderless, including ingestion, consumer offset, and metadata topics. The incident caused full ingestion outages for hours and then a prolonged degraded state, especially for Activity Log data, while responders tried to preserve offsets, restore ingest, prevent disk exhaustion, and recover damaged partitions. As administrative behavior in Kafka worsened over the following week, the team shifted from in-place recovery to planning and executing a controlled evacuation to a new cluster.

How to investigate an IT incident

  1. 01Symptom

    Record what users and systems actually reported.

  2. 02Hypothesis

    Keep the first plausible explanation provisional.

  3. 03Evidence

    Find observations that narrow the fault domain.

  4. 04Turning point

    Identify what changed the investigation’s direction.

  5. 05Root cause

    Name the mechanism that survived the evidence.

Explore the troubleshooting field guide →

Previous investigations

Which incidents are in the Casebook?

Full case archive →
  1. 02
    Clerk System Outage: PostgreSQL Query Plan Flip Triggered by Auto-ANALYZE Sampling Statistics ErrorFebruary 19, 2026 · Cloud Infrastructure
    7/10
  2. 03
    Resend outage on February 15, 2026: database connection exhaustion across mixed deployment patternsFebruary 16, 2026 · Applications
    7/10
  3. 04
    Supabase us-east-2 Regional Outage: Monitoring-Service Deployment Accidentally Enabled VPC Block Public Access Region-WideFebruary 12, 2026 · Networking
    6/10
  4. 05
    Redocly January 2026 service disruptions from orchestration-layer instability and queue retry cascadeJanuary 27, 2026 · Cloud Infrastructure
    8/10

Questions

Questions about the Casebook

What is Root Cause Casebook?
An evidence-grounded field guide to how technology teams diagnose public IT incidents. Each case preserves the hypotheses, wrong turns, decisive tests, and reasoning that narrowed the fault domain.
Which incidents does the Casebook cover?
Public technology incidents with enough public evidence to reconstruct the investigation: cloud outages, AI infrastructure degradations, and major service disruptions. Cases include the May 2026 Azure OpenAI multi-region failures, the June 2026 Google Gemini outage, and the November 2025 Cloudflare outage.
How are the cases researched?
Every case is reconstructed from public sources: status pages, official postmortems, and engineering write-ups. The methodology page documents the evidence standards, and uncertain claims are labeled as such instead of presented as fact.
Is Root Cause Casebook free?
Yes. All cases are free to read with no account required, and new investigations are published regularly.