By Steve Thoms · Published October 15, 2026 · Confidence: high · Complexity 9/10
Honeycomb EU Kafka outage after DR cleanup triggered leaderless partitions and forced cluster evacuation
After a December 5, 2025 disaster recovery exercise in Honeycomb's EU region, production cleanup steps intentionally destroyed Kafka brokers and left multiple partitions leaderless, including ingestion, consumer offset, and metadata topics. The incident caused full ingestion outages for hours and then a prolonged degraded state, especially for Activity Log data, while responders tried to preserve offsets, restore ingest, prevent disk exhaustion, and recover damaged partitions. As administrative behavior in Kafka worsened over the following week, the team shifted from in-place recovery to planning and executing a controlled evacuation to a new cluster.
Problem statement
After the December 5, 2025 DR exercise cleanup in Honeycomb's EU region, purposeful Kafka broker destruction left multiple topic partitions fully leaderless, causing ingestion outages and forcing a full cluster evacuation.
What investigators first believed
- The initial fault domain was ambiguous from the top-level symptom alone.
How the diagnosis unfolded
- 01
Review the source investigation
The source contained a troubleshooting sequence that should be reviewed and structured by an analyst before publication.
- 02
Collect supporting evidence
The analyst should extract the key observations, comparisons, or telemetry that materially narrowed the diagnosis.
- 03
Confirm restoration path
The analyst should verify how the source describes mitigation, validation, and recovery sequencing before publication.
What narrowed the fault domain
Source narrative
The source contains enough incident detail for analyst review, but the evidence structure should be refined before publication.
Reported impact
The externally visible symptom should be confirmed and sharpened during analyst review before publishing the case.
Key turning points
- Marking damaged partitions read-only on the producer side stopped Shepherd's crash loop without destructive repair.
- Disk usage kept rising with ingest off, disproving that tiered-storage cleanup was progressing; turning tiered storage off instantly restored disk.
- Repairing _confluent-tier-state and __consumer_offsets unblocked SLO processing (18h of late data started catching up), exposing hidden metadata dependency.
- When Kafka admin operations began timing out, responders split into cluster-salvage vs full-evacuation planning lanes.
Root cause
During cleanup of a December 5, 2025 disaster-recovery exercise in Honeycomb's EU region, the runbook step of purposefully destroying Kafka brokers — which had been safe in every pre-production run — destroyed older in-sync brokers in production and left multiple topic partitions with no live replicas (fully leaderless), damaging ingest topics, consumer-offset topics, and tiered-storage metadata topics (_confluent-tier-state, __consumer_offsets).
Resolution
Responders split into cluster-salvage and full-evacuation planning lanes; damaged partitions were marked read-only producer-side to stop the Shepherd ingest crash loop, Kafka autobalancing was disabled for manual reassignments, broken metadata topics were repaired, and tiered storage was turned off to restore disk headroom. Ingest was reopened after tiered storage was disabled, then the team evacuated to a brand-new Kafka cluster with a Retriever offset-reset mechanism; migration completed in about two hours.
Lessons from the response
- A recovery step that is safe in exercises is not automatically safe in production; cluster topology and data criticality differ.
- Model the full dependency chain: broken metadata and offset topics blocked administrative and storage behavior beyond the data path.
- Use negative evidence aggressively — disk usage rising with ingest off falsified the 'cleanup is progressing' assumption.
Troubleshooting principles
- 01
Do not trust a recovery step just because it was safe in exercises; production state can invalidate the same move.
- 02
Model the full dependency chain, including metadata planes and offset-tracking paths, not just the primary data path.
- 03
When automatic balancing or cleanup behaves opaquely during storage stress, switch to explicit manual control and narrower changes.
- 04
Contain secondary amplifiers early; stabilizing the crashing ingest tier bought time for deeper diagnosis.
- 05
Use negative evidence aggressively: if expected disk drain or reassignment progress does not appear, treat that as evidence against the current theory.
- 06
A repair that instantly changes downstream behavior is strong causal evidence; use those moments to redraw the incident map.
The case involves enough ambiguity and evidence collection to teach a reusable troubleshooting process.
Questions answered
What happened in the Honeycomb EU Kafka outage after DR cleanup triggered leaderless partitions and forced cluster evacuation incident?
After the December 5, 2025 DR exercise cleanup in Honeycomb's EU region, purposeful Kafka broker destruction left multiple topic partitions fully leaderless, causing ingestion outages and forcing a full cluster evacuation.
What was the root cause?
During cleanup of a December 5, 2025 disaster-recovery exercise in Honeycomb's EU region, the runbook step of purposefully destroying Kafka brokers — which had been safe in every pre-production run — destroyed older in-sync brokers in production and left multiple topic partitions with no live replicas (fully leaderless), damaging ingest topics, consumer-offset topics, and tiered-storage metadata topics (_confluent-tier-state, __consumer_offsets).
How was the root cause discovered?
Marking damaged partitions read-only on the producer side stopped Shepherd's crash loop without destructive repair. Disk usage kept rising with ingest off, disproving that tiered-storage cleanup was progressing; turning tiered storage off instantly restored disk. Repairing _confluent-tier-state and __consumer_offsets unblocked SLO processing (18h of late data started catching up), exposing hidden metadata dependency. When Kafka admin operations began timing out, responders split into cluster-salvage vs full-evacuation planning lanes.
What evidence mattered most?
The source contains enough incident detail for analyst review, but the evidence structure should be refined before publication. The externally visible symptom should be confirmed and sharpened during analyst review before publishing the case.
Which assumptions were wrong?
The approved source does not identify a specific incorrect assumption.
What delayed recovery?
The approved record describes restoration as follows: Responders split into cluster-salvage and full-evacuation planning lanes; damaged partitions were marked read-only producer-side to stop the Shepherd ingest crash loop, Kafka autobalancing was disabled for manual reassignments, broken metadata topics were repaired, and tiered storage was turned off to restore disk headroom. Ingest was reopened after tiered storage was disabled, then the team evacuated to a brand-new Kafka cluster with a Retriever offset-reset mechanism; migration completed in about two hours. It does not separately quantify a recovery delay unless stated in that account.
What should operators learn from this case?
A recovery step that is safe in exercises is not automatically safe in production; cluster topology and data criticality differ. Model the full dependency chain: broken metadata and offset topics blocked administrative and storage behavior beyond the data path. Use negative evidence aggressively — disk usage rising with ingest off falsified the 'cleanup is progressing' assumption.
Apply the diagnostic method
Use the Troubleshooting Field Guide to compare this investigation with the evidence patterns, hypothesis tests, and turning points found across the Casebook.
Open the Troubleshooting Field Guide →Original incident source
Root Cause Casebook separates reported facts from analyst synthesis. This public record was explicitly approved before export.