Confidence: high · Complexity 7/10
Cloudflare .de DNSSEC outage response
Broken DNSSEC signatures at the .de TLD caused validating resolvers to return SERVFAIL for fresh lookups. Cloudflare had to distinguish cached success from fresh-resolution failure, account for retry-driven traffic inflation, and decide whether to bypass DNSSEC validation temporarily to restore service.
Problem statement
Users resolving .de domains through 1.1.1.1 saw failures once cached records expired because fresh responses from DENIC contained invalid DNSSEC signatures.
What investigators first believed
- The top-level SERVFAIL symptom could be interpreted as generic upstream reachability trouble.
- Serve-stale behavior and overall NOERROR volume could make the incident appear less severe than fresh lookups actually were.
How the diagnosis unfolded
- 01
Compare resolver response codes beginning at incident start.
Cloudflare saw a SERVFAIL spike starting around 19:30 UTC on May 5 as validating lookups began failing.
- 02
Separate cached responses from fresh-resolution behavior.
Investigators confirmed that serve-stale was masking part of the user impact while fresh resolutions were failing.
- 03
Inspect Extended DNS Error behavior and DNSSEC validation details.
They found 1.1.1.1 was surfacing EDE 22 instead of a DNSSEC-specific error, which pointed at the wrong layer.
- 04
Assess mitigation options for a parent-zone signing failure.
The team chose an override equivalent to a Negative Trust Anchor to treat .de as insecure temporarily.
- 05
Coordinate mitigation rollout and operator communication.
The mitigation was rolled out at 22:17 UTC and shared with peer DNS operators while DENIC fixed the zone.
What narrowed the fault domain
Resolver response-code trend
SERVFAIL climbed over several hours as cached .de records expired.
Query volume inflation
Retry traffic inflated raw query counts, making volume look worse than unique user impact.
DNSSEC validation and EDE behavior
Broken RRSIGs caused validation failure, but 1.1.1.1 surfaced EDE 22 instead of DNSSEC Bogus.
Serve-stale cache behavior
Cached responses continued serving successfully even while fresh lookups failed.
Treating the EDE output as authoritative would have kept attention on connectivity rather than DNSSEC validation.
Key turning points
- Separating stale-cache success from fresh-resolution failure narrowed the fault domain quickly.
- Recognizing that retries were inflating demand clarified that raw volume was not equal to raw impact.
- Choosing an NTA-equivalent override reframed the response as a tradeoff between validation purity and service restoration.
Root cause
DENIC published incorrect DNSSEC signatures for the .de zone, forcing validating resolvers to reject fresh answers.
Resolution
Cloudflare temporarily marked .de as insecure on the relevant resolvers, restoring service while DENIC corrected the signing issue.
Lessons from the response
- Protocol-level evidence is more reliable than generic resolver error labels.
- Cache-hit behavior can hide the true shape of impact during DNS incidents.
- Shared-infrastructure incidents need pre-approved exception paths such as NTAs.
Troubleshooting principles
- 01
Separate preserved behavior from failed behavior early.
- 02
During shared infrastructure incidents, preplanned exception policies accelerate recovery.
The incident crossed protocol validation, resolver behavior, cache semantics, and operational tradeoffs under Internet-scale demand, but the root mechanism became clear once the DNSSEC layer was isolated.
Questions answered
What happened in the Cloudflare .de DNSSEC outage response incident?
Users resolving .de domains through 1.1.1.1 saw failures once cached records expired because fresh responses from DENIC contained invalid DNSSEC signatures.
What was the root cause?
DENIC published incorrect DNSSEC signatures for the .de zone, forcing validating resolvers to reject fresh answers.
How was the incident resolved?
Cloudflare temporarily marked .de as insecure on the relevant resolvers, restoring service while DENIC corrected the signing issue.
Original incident source
Root Cause separates reported facts from analyst synthesis. This public record was explicitly approved before export.