Queue Overloaded

Summary

Queue Overloaded

This incident has been resolved.

• Started

  Jun 11, 2026 · 3:37 PM

• Resolved

  Jun 11, 2026 · 8:49 PM

• Duration

  5h 12m

• Severity

  Minor

• ◆

  Identified

  Jun 11 · 3:37 PM Blue Canvas

  A queue that processes background jobs has become overloaded, and is affecting the creation of deployment requests. We're currently working to resolve this issue.

• ◉

  Monitoring

  Jun 11 · 4:23 PM Blue Canvas

  We identified the root cause. One of our queued jobs, scheduled to run every 10 seconds, began taking several minutes to complete. These long-running jobs accumulated and consumed all available worker capacity. We've deployed two fixes: - Concurrency limit, only one instance of this job can run at a time, preventing pile-up. - Rewritten algorithm, the job now resolves in ~1 second instead of minutes. We're continuing to monitor.

• ✓

  Resolved

  Jun 11 · 8:49 PM Blue Canvas

  This incident has been resolved.