Webhook APIs and UI Degraded

Summary

Webhook APIs and UI Degraded

On May 28, 2026, between 00:54 UTC and 01:19 UTC, some users experienced errors when interacting with the Webhooks API, including webhook delivery history and configuration endpoints. On average, the error rate was 0.28% and peaked at 0.45%. This was due to a bug that caused a single Kubernetes pod to enter a CrashLoopBackOff after receiving a 500 with an empty response body from Cosmos DB.<br /><br />We mitigated the incident by restarting the service. To prevent future incidents, we are pushing a change to handle this response scenario from Cosmos DB appropriately.

• Started

  May 28, 2026 · 1:13 AM

• Resolved

  May 28, 2026 · 1:32 AM

• Duration

  18m

• Severity

  Minor

• ◐

  Investigating

  May 28 · 1:13 AM GitHub

  We are investigating reports of degraded performance for Webhooks

• ◉

  Monitoring

  May 28 · 1:27 AM GitHub

  The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.

• ✓

  Resolved

  May 28 · 1:32 AM GitHub

  On May 28, 2026, between 00:54 UTC and 01:19 UTC, some users experienced errors when interacting with the Webhooks API, including webhook delivery history and configuration endpoints. On average, the error rate was 0.28% and peaked at 0.45%. This was due to a bug that caused a single Kubernetes pod to enter a CrashLoopBackOff after receiving a 500 with an empty response body from Cosmos DB.<br /><br />We mitigated the incident by restarting the service. To prevent future incidents, we are pushing a change to handle this response scenario from Cosmos DB appropriately.