Outage impacting web experience and pipelines starting

Summary

Outage impacting web experience and pipelines starting

## Summary On July 2, 2026, from 15:12 UTC to 16:44 UTC, some CircleCI customers experienced failures starting pipelines and difficulty accessing the CircleCI UI, including an inability to log in. Customers triggering pipelines via the API may also have had pipelines fail to run. The incident originated from a combination of internal maintenance and customer-initiated project deletions running concurrently, which caused elevated load on a data service. The connection layer between our API and that data service was not configured to time out slow responses under the elevated load conditions seen during this incident. This eventually exhausted available threads and prevented the request routing layer from handling any other requests, resulting in customer-facing errors. The affected services were scaled up and the internal maintenance job was canceled to reduce pressure, allowing all affected functionality to return to normal. Customers whose jobs failed during this window may rerun affected jobs. A small number of customers may still see workflows that appear to be running; this is actively being corrected and does not impact billing. We thank you for your patience while our team worked on implementing a fix. ## What Happened \(all times UTC\) At 13:30 on July 2, 2026, as part of normal platform maintenance, our engineering team began a data deletion job, divided into several hundred individual maintenance tasks. At 14:40, unrelated to our platform maintenance, a significant number of customer project deletion requests were received and began executing. At 15:12, the combined load from the maintenance tasks and the customer-driven projection deletions caused an internal data service to slow down, and this began cascading into our API layer. Because connections between the API layer and the workflows data service were not configured to time out slow responses under elevated load conditions, the API-handling components became overwhelmed, went unresponsive, and began failing their health checks. This caused our request routing layer to lose available upstream requests, resulting in customers seeing errors when attempting to access the UI, to log in via GitHub or Bitbucket OAuth, or to start pipelines. At 15:16, our monitoring systems alerted and engineers were paged. The team investigated and confirmed the source of the elevated load. At 15:30 Engineers began to scale up the affected services and cancelled a long-running maintenance task to drain the load. The team noted signs of that the system was recovering around 16:12 and adjusted the status to monitoring at 16:24. At 16:44, the team confirmed that all customer-facing services had recovered to normal operating levels. Background load from the deletion job drained by approximately 17:00. Following the incident, we discovered a small number of workflows continued to inaccurately report that they were in a running state. This should be corrected within the next few days. In all cases, customers will only be billed for actual compute time used. ## Future Prevention and Process Improvement Our team is implementing the following improvements as a result of this incident: * After the incident was resolved we added timeout protections on connections between our API layer and internal data services, so that slow responses from a downstream service cannot exhaust API worker capacity. * We are actively investigating ways to enforce resource limits on background data tasks to prevent them from impacting customer-facing services. * We are improving observability and alerting on the affected data service so that resource pressure is detected earlier, before it reaches a level that impacts customers.

• Started

  Jul 2, 2026 · 3:25 PM

• Resolved

  Jul 2, 2026 · 4:44 PM

• Duration

  1h 18m

• Severity

  Critical

• ◐

  Investigating

  Jul 2 · 3:25 PM CircleCI

  We are currently investigating this issue.

• ◐

  Investigating

  Jul 2 · 4:07 PM CircleCI

  We are investigating an issue that may cause some customers to be redirected to a login page and prevents some pipelines from starting. Thank you for your patience as we continue our mitigation of this issue.

• ◉

  Monitoring

  Jul 2 · 4:24 PM CircleCI

  We have identified a source of increased system load which caused failures processing API requests. We have mitigated the issue and jobs are now running as expected and UI access has recovered. Some pipelines may have failed or failed to start. Resubmitting should correct these issues. We are continuing to monitor to ensure recovery.

• ◉

  Monitoring

  Jul 2 · 4:29 PM CircleCI

  Pipelines are continuing to run as normal. We will continue monitoring for recovery.

• ✓

  Resolved

  Jul 2 · 4:44 PM CircleCI

  Pipelines and UI access have fully recovered. Please contact customer support if you have issues running builds. This incident has been resolved.

• ✓

  Postmortem

  Jul 10 · 6:57 PM CircleCI

  ## Summary On July 2, 2026, from 15:12 UTC to 16:44 UTC, some CircleCI customers experienced failures starting pipelines and difficulty accessing the CircleCI UI, including an inability to log in. Customers triggering pipelines via the API may also have had pipelines fail to run. The incident originated from a combination of internal maintenance and customer-initiated project deletions running concurrently, which caused elevated load on a data service. The connection layer between our API and that data service was not configured to time out slow responses under the elevated load conditions seen during this incident. This eventually exhausted available threads and prevented the request routing layer from handling any other requests, resulting in customer-facing errors. The affected services were scaled up and the internal maintenance job was canceled to reduce pressure, allowing all affected functionality to return to normal. Customers whose jobs failed during this window may rerun affected jobs. A small number of customers may still see workflows that appear to be running; this is actively being corrected and does not impact billing. We thank you for your patience while our team worked on implementing a fix. ## What Happened \(all times UTC\) At 13:30 on July 2, 2026, as part of normal platform maintenance, our engineering team began a data deletion job, divided into several hundred individual maintenance tasks. At 14:40, unrelated to our platform maintenance, a significant number of customer project deletion requests were received and began executing. At 15:12, the combined load from the maintenance tasks and the customer-driven projection deletions caused an internal data service to slow down, and this began cascading into our API layer. Because connections between the API layer and the workflows data service were not configured to time out slow responses under elevated load conditions, the API-handling components became overwhelmed, went unresponsive, and began failing their health checks. This caused our request routing layer to lose available upstream requests, resulting in customers seeing errors when attempting to access the UI, to log in via GitHub or Bitbucket OAuth, or to start pipelines. At 15:16, our monitoring systems alerted and engineers were paged. The team investigated and confirmed the source of the elevated load. At 15:30 Engineers began to scale up the affected services and cancelled a long-running maintenance task to drain the load. The team noted signs of that the system was recovering around 16:12 and adjusted the status to monitoring at 16:24. At 16:44, the team confirmed that all customer-facing services had recovered to normal operating levels. Background load from the deletion job drained by approximately 17:00. Following the incident, we discovered a small number of workflows continued to inaccurately report that they were in a running state. This should be corrected within the next few days. In all cases, customers will only be billed for actual compute time used. ## Future Prevention and Process Improvement Our team is implementing the following improvements as a result of this incident: * After the incident was resolved we added timeout protections on connections between our API layer and internal data services, so that slow responses from a downstream service cannot exhaust API worker capacity. * We are actively investigating ways to enforce resource limits on background data tasks to prevent them from impacting customer-facing services. * We are improving observability and alerting on the affected data service so that resource pressure is detected earlier, before it reaches a level that impacts customers.