Elevated API Errors

Summary

Elevated API Errors

# Elevated API Errors **Date:** 09/18/2026 **Duration:** 22:40–23:03 UTC \(23 minutes\) **Status:** Resolved ## Summary On September 18, 2026, from 22:40 to 23:03 UTC, WorkOS experienced an outage affecting authentication and other API requests. Customers encountered failed sign-ins, API errors, timeouts, and long delays. Authentication was most severely affected. An infrastructure failover in the cache supporting our feature-flag system exposed weaknesses in application connection recovery and fallback behavior. Some connections remained stalled after the cache was available again, delaying requests and exhausting database connection pools. Service recovered by 23:03 UTC. We subsequently deployed changes to improve connection recovery. We’re sorry for the disruption this caused you and your users. ## Background WorkOS uses feature flags to control application behavior. The affected backend services read flag data from a shared Redis datastore populated by LaunchDarkly Relay. Each application also keeps a local cache, with a 30-second expiration. While an entry is valid, the application can evaluate it locally; after it expires, the application reads Redis. The Redis cluster already had a primary and replica in separate AWS availability zones, with automatic failover enabled. This redundancy allowed AWS to promote a replacement primary. Applications still needed to detect failed connections and reconnect to that primary. ## What Happened ### Affected products Some requests experienced errors or delays across the following products: * **AuthKit:** Sign-in, token requests, user-management operations, and hosted authentication pages. * **Single Sign-On \(SSO\):** Authorization and SAML callback requests. * **WorkOS API:** Selected operations, including organization requests, API-key validation, and connection requests. * **Dashboard:** Some operations that relied on the affected APIs. * **Admin Portal:** Portal link generation and monitored setup requests. * **Directory Sync:** SCIM OAuth token requests, plus delays and client disconnects in Directory Sync API traffic. * **Audit Logs:** Management operations, particularly action-schema creation. ### Customer impact Customer impact lasted from **22:40 to 23:03 UTC**. Across our public edge, **24.1% of recorded requests returned server errors**, and **17.2% ended with the client disconnecting**. The table breaks down request groups by product. | Product | Server errors \(HTTP 5xx\) | Client disconnects \(HTTP 499\) | | --- | --- | --- | | AuthKit, User Management and organization APIs | 37.22% | 24.59% | | Hosted AuthKit \(default-domain proxy\) | 7.27% | 13.77% | | SSO and connection APIs | 2.22% | 2.31% | | Dashboard | 27.10% | 8.08% | | Admin Portal | 24.61% | 7.47% | | Directory Sync APIs | 0.08% | 2.80% | | Audit Logs management APIs | 22.82% | 3.55% | | Audit Logs event capture | 0.0006% | 0.11% | #### Latency Samples * **Authenticate POST requests:** At the peak, one-minute median latency reached approximately 32 seconds and p95 latency reached approximately 45 seconds, compared with roughly 65 ms and 140 ms before the incident. * **Directory Sync:** Peak one-minute p95 latency for selected SCIM user-creation and PATCH-update operations reached approximately 22 seconds, compared with roughly 65 ms before the incident. ### Root cause and contributing factors Around 22:40 UTC, the Redis primary became unavailable to application clients. AWS ElastiCache promoted its replica, completing the failover at 22:40:17 UTC. New connections could reach the promoted node, but some existing application connections remained stalled. Three application behaviors prolonged and amplified the interruption: 1. **Flag-call timeouts did not recover stalled connections.** An application-level deadline returned a fallback value after two seconds, but it did not cancel the underlying Redis command or replace the connection. Additional reads accumulated while clients waited to detect the connection failure. Some commands remained pending for approximately 16 minutes. 2. **External waits held database connections open.** Some flag checks occurred inside database transactions. Repeated waits of several seconds kept pooled database connections occupied and, in some cases, held row locks. Available connections were consumed, causing other requests to queue and fail. Database compute capacity was not the limiting factor. 3. **A fallback selected an incompatible decryption path.** A flag used during a decryption-service migration defaulted to the old behavior when flag evaluation failed. That sent some encrypted values to a legacy service that could not decrypt them, causing additional authentication failures. The local cache reduced normal Redis traffic, but expired entries still depended on Redis. Our two-second deadline limited how long a flag caller waited; it did not provide prompt recovery of a stale, established connection. This was a gap in our resiliency safeguards. ### Recovery and communication Responders restarted affected application instances, which triggered reconnection to the healthy Redis instance. As connections recovered and queued work drained, error rates and latency returned to normal. We also identified a notification gap: the initial status page update did not select the affected service components. Customers subscribed only to those components therefore did not receive the incident notification. ## Timeline All times are UTC. * **September 18, 22:40** — Customer-facing errors and latency increase. AWS completes promotion of the Redis replica at 22:40:17. * **22:44** — Engineers declare an incident following automated alerts. * **22:49** — The public status-page incident is opened. * **22:53** — Investigation identifies stalled feature-flag Redis clients and database connection starvation. * **22:55–23:03** — Responders restart affected application instances, existing clients begin reconnecting, and queued work drains. Recovery progresses across services. * **23:03** — Customer-facing error rates and latency return to normal. * **23:29** — The status-page incident is marked resolved after monitoring. * **September 19, 01:44** — A Redis-client recovery fix is deployed to production after validation with controlled failure and failover tests in staging. ## Remediation ### Immediate Fixes Completed: * **Restored service:** Restarted affected application instances and monitored recovery as clients reconnected and database queues drained. * **Bounded Redis failures:** Deployed a Redis-client change to detect stalled established connections, bound failed operations, and reconnect more promptly. * **Corrected the decryption fallback:** Changed the migration flag’s fallback to behavior compatible with both current and legacy ciphertext formats. * **Validated the recovery fix:** Tested stalled-client behavior and an ElastiCache failover in staging before deploying the connection changes to production. ### Near-term Improvements The following work remains open: * Extend bounded connection recovery to other Redis clients and verify their behavior during failover \(in progress\). * Remove feature-flag and decryption-service network waits from authentication database transactions \(in progress\). * Complete the legacy decryption migration and retire its obsolete flag and fallback paths. * Audit feature-flag defaults against stable production behavior and retire flags whose migrations are complete. * Evaluate and pilot streaming flag updates into application memory, reducing the need to contact Redis while serving requests. ### Preventive Measures Our follow-up work includes: * Improving visibility into flag-evaluation timeouts, fallback use, and connection recovery. * Adding checks for external network waits inside database transactions. * Requiring affected service components to be selected when opening a status-page incident, so component-specific subscribers receive notifications. ## Conclusion We recognize the impact this incident had on you and your customers. We’re sorry for the interruption and for the notification gap. We have deployed the immediate connection-recovery and fallback fixes. We will track the remaining corrective work through completion and update this analysis as unresolved findings are confirmed.

• Started

  Sep 18, 2026 · 6:49 PM

• Resolved

  Sep 18, 2026 · 7:29 PM

• Duration

  39m

• Severity

  Major

• ◐

  Investigating

  Sep 18 · 6:49 PM WorkOS

  We are investigating an issue with our API. We apologize for the inconvenience and will share an update once we have more information.

• ◐

  Investigating

  Sep 18 · 7:01 PM WorkOS

  We're continuing to investigate this issue to identify the root cause. We'll share another update soon.

• ◆

  Identified

  Sep 18 · 7:05 PM WorkOS

  We've identified the root cause, and we're seeing services are beginning to recover. We'll continue to monitor closely and share updates.

• ◉

  Monitoring

  Sep 18 · 7:09 PM WorkOS

  Our initial remediations are in place, and services have recovered. We're continuing to monitor closely.

• ✓

  Resolved

  Sep 18 · 7:29 PM WorkOS

  Our remediations continue to be effective, and services continue to be recovered. We will continue further investigation and additional preventative measures.

• ✓

  Postmortem

  Sep 21 · 8:53 PM WorkOS

  # Elevated API Errors **Date:** 09/18/2026 **Duration:** 22:40–23:03 UTC \(23 minutes\) **Status:** Resolved ## Summary On September 18, 2026, from 22:40 to 23:03 UTC, WorkOS experienced an outage affecting authentication and other API requests. Customers encountered failed sign-ins, API errors, timeouts, and long delays. Authentication was most severely affected. An infrastructure failover in the cache supporting our feature-flag system exposed weaknesses in application connection recovery and fallback behavior. Some connections remained stalled after the cache was available again, delaying requests and exhausting database connection pools. Service recovered by 23:03 UTC. We subsequently deployed changes to improve connection recovery. We’re sorry for the disruption this caused you and your users. ## Background WorkOS uses feature flags to control application behavior. The affected backend services read flag data from a shared Redis datastore populated by LaunchDarkly Relay. Each application also keeps a local cache, with a 30-second expiration. While an entry is valid, the application can evaluate it locally; after it expires, the application reads Redis. The Redis cluster already had a primary and replica in separate AWS availability zones, with automatic failover enabled. This redundancy allowed AWS to promote a replacement primary. Applications still needed to detect failed connections and reconnect to that primary. ## What Happened ### Affected products Some requests experienced errors or delays across the following products: * **AuthKit:** Sign-in, token requests, user-management operations, and hosted authentication pages. * **Single Sign-On \(SSO\):** Authorization and SAML callback requests. * **WorkOS API:** Selected operations, including organization requests, API-key validation, and connection requests. * **Dashboard:** Some operations that relied on the affected APIs. * **Admin Portal:** Portal link generation and monitored setup requests. * **Directory Sync:** SCIM OAuth token requests, plus delays and client disconnects in Directory Sync API traffic. * **Audit Logs:** Management operations, particularly action-schema creation. ### Customer impact Customer impact lasted from **22:40 to 23:03 UTC**. Across our public edge, **24.1% of recorded requests returned server errors**, and **17.2% ended with the client disconnecting**. The table breaks down request groups by product. | Product | Server errors \(HTTP 5xx\) | Client disconnects \(HTTP 499\) | | --- | --- | --- | | AuthKit, User Management and organization APIs | 37.22% | 24.59% | | Hosted AuthKit \(default-domain proxy\) | 7.27% | 13.77% | | SSO and connection APIs | 2.22% | 2.31% | | Dashboard | 27.10% | 8.08% | | Admin Portal | 24.61% | 7.47% | | Directory Sync APIs | 0.08% | 2.80% | | Audit Logs management APIs | 22.82% | 3.55% | | Audit Logs event capture | 0.0006% | 0.11% | #### Latency Samples * **Authenticate POST requests:** At the peak, one-minute median latency reached approximately 32 seconds and p95 latency reached approximately 45 seconds, compared with roughly 65 ms and 140 ms before the incident. * **Directory Sync:** Peak one-minute p95 latency for selected SCIM user-creation and PATCH-update operations reached approximately 22 seconds, compared with roughly 65 ms before the incident. ### Root cause and contributing factors Around 22:40 UTC, the Redis primary became unavailable to application clients. AWS ElastiCache promoted its replica, completing the failover at 22:40:17 UTC. New connections could reach the promoted node, but some existing application connections remained stalled. Three application behaviors prolonged and amplified the interruption: 1. **Flag-call timeouts did not recover stalled connections.** An application-level deadline returned a fallback value after two seconds, but it did not cancel the underlying Redis command or replace the connection. Additional reads accumulated while clients waited to detect the connection failure. Some commands remained pending for approximately 16 minutes. 2. **External waits held database connections open.** Some flag checks occurred inside database transactions. Repeated waits of several seconds kept pooled database connections occupied and, in some cases, held row locks. Available connections were consumed, causing other requests to queue and fail. Database compute capacity was not the limiting factor. 3. **A fallback selected an incompatible decryption path.** A flag used during a decryption-service migration defaulted to the old behavior when flag evaluation failed. That sent some encrypted values to a legacy service that could not decrypt them, causing additional authentication failures. The local cache reduced normal Redis traffic, but expired entries still depended on Redis. Our two-second deadline limited how long a flag caller waited; it did not provide prompt recovery of a stale, established connection. This was a gap in our resiliency safeguards. ### Recovery and communication Responders restarted affected application instances, which triggered reconnection to the healthy Redis instance. As connections recovered and queued work drained, error rates and latency returned to normal. We also identified a notification gap: the initial status page update did not select the affected service components. Customers subscribed only to those components therefore did not receive the incident notification. ## Timeline All times are UTC. * **September 18, 22:40** — Customer-facing errors and latency increase. AWS completes promotion of the Redis replica at 22:40:17. * **22:44** — Engineers declare an incident following automated alerts. * **22:49** — The public status-page incident is opened. * **22:53** — Investigation identifies stalled feature-flag Redis clients and database connection starvation. * **22:55–23:03** — Responders restart affected application instances, existing clients begin reconnecting, and queued work drains. Recovery progresses across services. * **23:03** — Customer-facing error rates and latency return to normal. * **23:29** — The status-page incident is marked resolved after monitoring. * **September 19, 01:44** — A Redis-client recovery fix is deployed to production after validation with controlled failure and failover tests in staging. ## Remediation ### Immediate Fixes Completed: * **Restored service:** Restarted affected application instances and monitored recovery as clients reconnected and database queues drained. * **Bounded Redis failures:** Deployed a Redis-client change to detect stalled established connections, bound failed operations, and reconnect more promptly. * **Corrected the decryption fallback:** Changed the migration flag’s fallback to behavior compatible with both current and legacy ciphertext formats. * **Validated the recovery fix:** Tested stalled-client behavior and an ElastiCache failover in staging before deploying the connection changes to production. ### Near-term Improvements The following work remains open: * Extend bounded connection recovery to other Redis clients and verify their behavior during failover \(in progress\). * Remove feature-flag and decryption-service network waits from authentication database transactions \(in progress\). * Complete the legacy decryption migration and retire its obsolete flag and fallback paths. * Audit feature-flag defaults against stable production behavior and retire flags whose migrations are complete. * Evaluate and pilot streaming flag updates into application memory, reducing the need to contact Redis while serving requests. ### Preventive Measures Our follow-up work includes: * Improving visibility into flag-evaluation timeouts, fallback use, and connection recovery. * Adding checks for external network waits inside database transactions. * Requiring affected service components to be selected when opening a status-page incident, so component-specific subscribers receive notifications. ## Conclusion We recognize the impact this incident had on you and your customers. We’re sorry for the interruption and for the notification gap. We have deployed the immediate connection-recovery and fallback fixes. We will track the remaining corrective work through completion and update this analysis as unresolved findings are confirmed.