Confidence: high · Complexity 9/10
Microsoft Azure OpenAI multi-region latency and failures PIR
Azure OpenAI experienced multi-region latency and failures after an upstream API change caused retry amplification from an internal Microsoft 365 workload. The investigation initially followed a credible but wrong crash signature before a second regional wave revealed the real mechanism.
Problem statement
Customers saw increased latency, intermittent failures, timeouts, and HTTP 5XX responses between 09:39 UTC and 17:05 UTC on May 29, 2026.
What investigators first believed
- Early crash signatures suggested an internal error-handling feature was causing instability.
- Improvement after disabling that feature appeared to confirm the diagnosis.
How the diagnosis unfolded
- 01
Detect falling success rates and correlate failures across regions and models.
Azure established that the incident was broad and affected shared routing behavior.
- 02
Inspect early crash diagnostics in the first impacted region, Australia East.
Investigators formed a plausible theory around an internal feature tied to error handling.
- 03
Disable the suspected feature.
Service health improved as traffic also naturally declined, creating a false sense of confirmation.
- 04
Observe recurrence when the upstream rollout expanded to Sweden Central.
The same failure pattern reappeared at greater scale, invalidating the first explanation.
- 05
Refocus on upstream retry behavior and isolate amplified traffic.
The team identified retry amplification from an internal first-party workload, isolated it, and coordinated rollback of the triggering change.
What narrowed the fault domain
Request success rates and latency
Automated monitoring showed a broad drop in request success and elevated latency across regions.
Australia East versus Sweden Central recurrence
The second wave in Sweden Central exposed the earlier mitigation as incomplete and highlighted the load-sensitive retry mechanism.
Inference routing layer resource exhaustion
Amplified internal retry traffic exhausted shared routing resources and caused out-of-memory crashes.
Upstream API-layer rollout history
The trigger was an upstream change that altered how capacity failures surfaced to callers.
Disabling the suspected internal feature addressed a secondary symptom and briefly looked like the fix.
Key turning points
- The Sweden Central recurrence invalidated the first explanation.
- Correlating the overload with upstream retry behavior separated the trigger from the misleading crash symptom.
- Identifying the first-party workload origin clarified why shared infrastructure protections failed.
Root cause
An upstream API-layer change caused capacity-related failures to surface as retriable server errors, creating a large retry storm from an internal Microsoft 365 workload that overwhelmed the shared Azure OpenAI routing layer.
Resolution
Azure isolated the offending workload onto dedicated infrastructure, coordinated rollback of the upstream change, reduced retry-driven traffic, and cleared backlogged requests.
Lessons from the response
- Symptom relief is not proof of diagnosis.
- Retry storms must be treated as a first-class incident mechanism.
- Recurrence under a different load profile can disprove an attractive theory quickly.
Troubleshooting principles
- 01
A convincing early theory is still provisional until it survives a second wave of evidence.
- 02
Trigger, amplifier, and visible failure path must be separated.
This case combines multi-region behavior, internal dependency contracts, retry amplification, shared multi-tenant infrastructure, and misleading symptom relief before the true cause became clear.
Questions answered
What happened in the Microsoft Azure OpenAI multi-region latency and failures PIR incident?
Customers saw increased latency, intermittent failures, timeouts, and HTTP 5XX responses between 09:39 UTC and 17:05 UTC on May 29, 2026.
What was the root cause?
An upstream API-layer change caused capacity-related failures to surface as retriable server errors, creating a large retry storm from an internal Microsoft 365 workload that overwhelmed the shared Azure OpenAI routing layer.
How was the incident resolved?
Azure isolated the offending workload onto dedicated infrastructure, coordinated rollback of the upstream change, reduced retry-driven traffic, and cleared backlogged requests.
Original incident source
Root Cause separates reported facts from analyst synthesis. This public record was explicitly approved before export.