Delays with Code Scanning and Billing

Summary

Delays with Code Scanning and Billing

Starting from 13:00 UTC June 1, 2026, to 00:17 UTC June 2, 2026, multiple services experienced delayed job processing due to increased latency in our background job queue service. The root cause was insufficient queue processing capacity to handle a large week-over-week increase in total job traffic.<br /><br />Users saw up to 90 minutes of delay in billing usage updates, 30 minutes of delay for webhook notifications to show, and 15 minutes of delay to see email notifications. Mitigation involved scaling up our background job service capacity to handle the spike in job traffic.<br /><br />We have added queue capacity monitoring to our background job queue service to stay ahead of weekly growth patterns and to reduce time to detect in the future.

• Started

  Jun 1, 2026 · 3:17 PM

• Resolved

  Jun 2, 2026 · 12:17 AM

• Duration

  9h

• Severity

  Minor

• ◐

  Investigating

  Jun 1 · 3:17 PM GitHub

  We are investigating reports of impacted performance for some GitHub services.

• ◐

  Investigating

  Jun 1 · 3:17 PM GitHub

  We are investigating reports of delayed code scanning runs and delayed billing updates.  We will continue to keep users updated on progress towards mitigation.

• ◐

  Investigating

  Jun 1 · 3:43 PM GitHub

  We are investigating reports of delayed code scanning runs, billing updates, email and mobile push notifications. We are investigating delays in our job processing architecture. We will continue to keep users updated on progress towards mitigation.

• ◐

  Investigating

  Jun 1 · 4:42 PM GitHub

  We are investigating reports of delayed billing updates, on GitHub.com. We are continuing to investigate delays in our job processing architecture. We are attempting to mitigate at the infrastructure level.  Code scanning runs and notifications have recovered.  We will continue to keep users updated on progress towards mitigation.<br /><br />GitHub Enterprise Cloud with Data Residency is not impacted.

• ◐

  Investigating

  Jun 1 · 5:28 PM GitHub

  We are continuing to investigate delayed billing updates, on GitHub.com.   We have applied multiple mitigations and are seeing some signs of improvement, and are continuing to work to improve the processing rate.  We will continue to keep users updated on progress towards mitigation.<br /><br />GitHub Enterprise Cloud with Data Residency is not impacted.

• ◐

  Investigating

  Jun 1 · 6:48 PM GitHub

  We are continuing to investigate delayed billing updates, on GitHub.com.  We have applied additional mitigations and are seeing some more signs of improvement, and are continuing to work to improve the processing rate. We will continue to keep users updated on progress towards mitigation.<br /><br />GitHub Enterprise Cloud with Data Residency is not impacted.

• ◐

  Investigating

  Jun 1 · 7:27 PM GitHub

  We are continuing to investigate delayed billing updates, on GitHub.com.  We have applied additional mitigations are continuing to see signs of improvement, and are continuing to work to improve the processing rate. We will continue to keep users updated on progress towards mitigation.<br /><br />GitHub Enterprise Cloud with Data Residency is not impacted.<br /><br />Next update in 2 hours.

• ◐

  Investigating

  Jun 1 · 9:59 PM GitHub

  We have identified the root cause and applied mitigations to address delays in billing updates and are continuing to see improvement in the processing rate. We will continue to monitor the progress and will provide an update in few hours.<br /><br />GitHub Enterprise Cloud with Data Residency is not impacted.<br />

• ✓

  Resolved

  Jun 2 · 12:17 AM GitHub

  Starting from 13:00 UTC June 1, 2026, to 00:17 UTC June 2, 2026, multiple services experienced delayed job processing due to increased latency in our background job queue service. The root cause was insufficient queue processing capacity to handle a large week-over-week increase in total job traffic.<br /><br />Users saw up to 90 minutes of delay in billing usage updates, 30 minutes of delay for webhook notifications to show, and 15 minutes of delay to see email notifications. Mitigation involved scaling up our background job service capacity to handle the spike in job traffic.<br /><br />We have added queue capacity monitoring to our background job queue service to stay ahead of weekly growth patterns and to reduce time to detect in the future.