QStash Service Disruption in the EU Region

Summary

QStash Service Disruption in the EU Region

## What happened A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until **11:24 UTC**; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers. QStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally. ## Customer impact Impact was limited to the EU region and to two effects within the window above: * **Delivery delays / failures** _\(09:11 – 09:45 UTC\)_ — messages dispatched through affected instances may have failed on their first attempt. Thanks to automatic retries with backoff, most were re-delivered once connectivity recovered; messages that exhausted their retry policy during the window followed their configured failure path \(e.g., DLQ / failure callback\). * **Allowlist rejections** _\(09:11 – 11:24 UTC\)_ — customers who restrict inbound traffic to QStash's outbound IP ranges may have seen requests from the affected servers rejected. Customers who do not enforce source-IP allowlisting were unaffected by this. ## Root cause A routine networking configuration change contained an error that: 1. disrupted outbound connectivity on the affected nodes, and 2. caused affected nodes to acquire outbound IPs outside the advertised range. The underlying gap was systemic: the change procedure had no automated validation gate to catch the faulty state before it reached production. ## What we've done * **Fail-fast pre-checks** — automated validation now halts the change procedure _before it runs_ if a configuration would degrade connectivity or violate expected state. * **Egress IP-range enforcement** — outbound IP assignments are now checked against the advertised QStash range, so a node can no longer come online with an unadvertised IP. * **Tighter post-change verification** — completion now confirms outbound reachability and egress-IP conformance across affected nodes. ## Customer action None required. All QStash traffic again originates from the advertised outbound IP ranges. We apologize for the disruption.

• Started

  Jun 19, 2026 · 9:11 AM

• Resolved

  Jun 19, 2026 · 9:11 AM

• Duration

  < 1m

• Severity

  Major

• ✓

  Resolved

  Jun 19 · 10:07 AM Upstash

  We identified an outbound networking issue affecting a subset of QStash server instances in the EU region between 2026-06-19 09:11 UTC and 2026-06-19 09:45 UTC. During this period, some requests dispatched by QStash may have failed. The issue has been resolved, and QStash is currently operating normally. We will publish a postmortem with additional details as soon as it is available. Update: We identified that one QStash server had been assigned an outbound IP address outside of the configured QStash IP range. This remained the case until 2026-06-19 11:24 UTC. Endpoints that restrict access based on QStash outbound IP allowlists may have rejected requests from this server during this period.

• ✓

  Postmortem

  Jun 19 · 1:52 PM Upstash

  ## What happened A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until **11:24 UTC**; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers. QStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally. ## Customer impact Impact was limited to the EU region and to two effects within the window above: * **Delivery delays / failures** _\(09:11 – 09:45 UTC\)_ — messages dispatched through affected instances may have failed on their first attempt. Thanks to automatic retries with backoff, most were re-delivered once connectivity recovered; messages that exhausted their retry policy during the window followed their configured failure path \(e.g., DLQ / failure callback\). * **Allowlist rejections** _\(09:11 – 11:24 UTC\)_ — customers who restrict inbound traffic to QStash's outbound IP ranges may have seen requests from the affected servers rejected. Customers who do not enforce source-IP allowlisting were unaffected by this. ## Root cause A routine networking configuration change contained an error that: 1. disrupted outbound connectivity on the affected nodes, and 2. caused affected nodes to acquire outbound IPs outside the advertised range. The underlying gap was systemic: the change procedure had no automated validation gate to catch the faulty state before it reached production. ## What we've done * **Fail-fast pre-checks** — automated validation now halts the change procedure _before it runs_ if a configuration would degrade connectivity or violate expected state. * **Egress IP-range enforcement** — outbound IP assignments are now checked against the advertised QStash range, so a node can no longer come online with an unadvertised IP. * **Tighter post-change verification** — completion now confirms outbound reachability and egress-IP conformance across affected nodes. ## Customer action None required. All QStash traffic again originates from the advertised outbound IP ranges. We apologize for the disruption.