QStash EU Region — Degraded Performance

Summary

QStash EU Region — Degraded Performance

### **Summary** On June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption under load, leading to reduced performance on affected shards. The incident was resolved by rolling back to the previous stable version. ### **Impact** * Service: QStash \(EU-CENTRAL-1\) * Start: 2026-06-23 16:22 UTC * Resolved: 2026-06-23 17:14 UTC * Duration: ~52 minutes * Impact: Increased latency and degraded performance for workloads routed to the affected instance in the EU. Multiple customers in the region experienced slower request processing during the incident window. ### **Root Cause** We recently deployed an enhancement to our Flow Control feature, designed to improve fairness between independent flow controls when unused global parallelism capacity was available. Under production load, the new behavior introduced a regression that significantly increased resource utilization. The elevated resource consumption caused performance degradation on the affected QStash instamce, impacting all workloads sharing that instance. ### **Resolution** After identifying the regression as the source of the slowdown, we rolled back the deployment to the last known stable version. * Rollback completed: 2026-06-23 19:18:46.84 UTC Following the rollback, system performance returned to normal levels and service stability was restored. ### **Preventive Actions** To reduce the likelihood of similar incidents in the future, we are taking the following actions: * Expand performance and load testing coverage for Flow Control changes. * Add resource utilization regression checks to the deployment pipeline. * Improve monitoring and alerting for abnormal CPU and memory consumption patterns. * Introduce additional canary validation before wider production rollout of scheduler-related changes. We apologize for the disruption and appreciate our customers’ patience while we resolved the issue.

• Started

  Jun 23, 2026 · 4:22 PM

• Resolved

  Jun 23, 2026 · 5:14 PM

• Duration

  52m

• Severity

  Minor

• ◐

  Investigating

  Jun 23 · 4:22 PM Upstash

  We are currently investigating this issue.

• ◉

  Monitoring

  Jun 23 · 4:32 PM Upstash

  A fix has been implemented and we are monitoring the results.

• ✓

  Resolved

  Jun 23 · 5:14 PM Upstash

  This incident has been resolved.

• ✓

  Postmortem

  Jun 24 · 12:28 PM Upstash

  ### **Summary** On June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption under load, leading to reduced performance on affected shards. The incident was resolved by rolling back to the previous stable version. ### **Impact** * Service: QStash \(EU-CENTRAL-1\) * Start: 2026-06-23 16:22 UTC * Resolved: 2026-06-23 17:14 UTC * Duration: ~52 minutes * Impact: Increased latency and degraded performance for workloads routed to the affected instance in the EU. Multiple customers in the region experienced slower request processing during the incident window. ### **Root Cause** We recently deployed an enhancement to our Flow Control feature, designed to improve fairness between independent flow controls when unused global parallelism capacity was available. Under production load, the new behavior introduced a regression that significantly increased resource utilization. The elevated resource consumption caused performance degradation on the affected QStash instamce, impacting all workloads sharing that instance. ### **Resolution** After identifying the regression as the source of the slowdown, we rolled back the deployment to the last known stable version. * Rollback completed: 2026-06-23 19:18:46.84 UTC Following the rollback, system performance returned to normal levels and service stability was restored. ### **Preventive Actions** To reduce the likelihood of similar incidents in the future, we are taking the following actions: * Expand performance and load testing coverage for Flow Control changes. * Add resource utilization regression checks to the deployment pipeline. * Improve monitoring and alerting for abnormal CPU and memory consumption patterns. * Introduce additional canary validation before wider production rollout of scheduler-related changes. We apologize for the disruption and appreciate our customers’ patience while we resolved the issue.