Incident with Git Operations, Pull Requests and Actions

Summary

Incident with Git Operations, Pull Requests and Actions

On October 7, 2026, GitHub had two outages of write operations with the same cause, totaling about 19 minutes. The first ran from 15:06 to 15:16 UTC (about 10 minutes) and the second from 16:52 to 17:01 UTC (about 9 minutes). In both windows, actions that save or change data failed across Git operations, pull requests, issues, GitHub Actions, the API, webhooks, sign-in and authentication, GitHub Apps and OAuth, code scanning, Codespaces, and Copilot. Reading data was largely unaffected. In the first outage, 5.6% of all web and API requests failed, peaking at 8.9%, and more than 12,000 organizations saw failed requests. Up to 26% of Git authentication requests failed, and 17% of Actions workflow runs failed with infrastructure errors. In the second outage, 6.2% of web and API requests failed, peaking at 9.3%, and more than 11,000 organizations were affected. Up to 25% of Git authentication requests failed, and 19% of Actions workflow runs failed. No databases experienced data loss, and operations retried after recovery succeeded. GitHub relies on an internal service that keeps track of which database servers should receive new and updated data. While replacing machines that run this service as part of regular maintenance, one of the new machines started with an outdated copy of its records, which still listed database servers we had already retired. When this node came online, it incorrectly modified routing configuration with the outdated records, routing write requests to non-existent database servers. Each outage ended when one of our established machines with correct configuration became the leader of the quorum and sent the correct configuration. After the first outage, we identified the new machine as the source and paused automated maintenance to stop it from being leader again. At 16:52 UTC, an automated maintenance task that did not honor the pause put it back in charge, and the outage happened again. We shut that machine down at 17:02 UTC and have since removed it, and the automation now honors the pause, which was a new regression due to an update of our puppet infrastructure. To prevent recurrence, we are clearing retired database servers from the service's records before any new machine is added, and we are keeping new machines from being elected leader until their records have been checked. We are also changing our systems to reject configuration that point to servers that don't exist and to keep using the last known-good settings instead. Finally, we are giving responders a faster way to take a misbehaving machine out of service, and adding alerts so changes like this are spotted within seconds. We will share a more detailed write-up of these incidents in our monthly availability report, along with the status of the repairs we are making to prevent issues like these in the future.

• Started

  Oct 7, 2026 · 3:14 PM

• Resolved

  Oct 7, 2026 · 4:25 PM

• Duration

  1h 10m

• Severity

  Critical

• ◐

  Investigating

  Oct 7 · 3:14 PM GitHub

  We are investigating reports of degraded availability for Actions, Git Operations and Pull Requests

• ◐

  Investigating

  Oct 7 · 3:14 PM GitHub

  Webhooks is experiencing degraded performance. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:15 PM GitHub

  Issues is experiencing degraded performance. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:16 PM GitHub

  Webhooks is experiencing degraded availability. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:21 PM GitHub

  Git Operations is experiencing degraded performance. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:24 PM GitHub

  We observed widespread impact across GitHub services between 15:06 UTC and 15:16 UTC, and are investigating the root cause. We are seeing all systems recover, and will provide another update shortly.

• ◐

  Investigating

  Oct 7 · 3:25 PM GitHub

  Pull Requests is experiencing degraded performance. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:27 PM GitHub

  Git Operations is operating normally.

• ◐

  Investigating

  Oct 7 · 3:27 PM GitHub

  Webhooks is experiencing degraded performance. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:31 PM GitHub

  The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.

• ◐

  Investigating

  Oct 7 · 3:32 PM GitHub

  Actions is operating normally.

• ◐

  Investigating

  Oct 7 · 3:35 PM GitHub

  Git Operations is experiencing degraded performance. We are continuing to investigate.

• ◐

  Investigating

  Oct 7 · 3:41 PM GitHub

  We are observing partial recovery across all systems. We are continuing to monitor and investigate the root cause of the disruption.

• ◐

  Investigating

  Oct 7 · 3:47 PM GitHub

  The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.

• ◐

  Investigating

  Oct 7 · 3:49 PM GitHub

  The degradation affecting Git Operations and Webhooks has been mitigated. We are monitoring to ensure stability.

• ◉

  Monitoring

  Oct 7 · 3:49 PM GitHub

  The degradation has been mitigated. We are monitoring to ensure stability.

• ◉

  Monitoring

  Oct 7 · 3:56 PM GitHub

  The degradation has been mitigated. We are monitoring to ensure stability.

• ◉

  Monitoring

  Oct 7 · 3:58 PM GitHub

  We are seeing full recovery across all systems including Git Operations, Pull Requests and Actions. We are continuing to investigate the root cause of the disruption between 15:06 UTC and 15:16 UTC, and will share another update shortly.

• ◉

  Monitoring

  Oct 7 · 4:23 PM GitHub

  Git Operations, Pull Requests and Actions are fully recovered and stable. The root cause has been identified and mitigations are being applied to prevent a reoccurrence.

• ✓

  Resolved

  Oct 7 · 4:25 PM GitHub

  On October 7, 2026, GitHub had two outages of write operations with the same cause, totaling about 19 minutes. The first ran from 15:06 to 15:16 UTC (about 10 minutes) and the second from 16:52 to 17:01 UTC (about 9 minutes). In both windows, actions that save or change data failed across Git operations, pull requests, issues, GitHub Actions, the API, webhooks, sign-in and authentication, GitHub Apps and OAuth, code scanning, Codespaces, and Copilot. Reading data was largely unaffected. In the first outage, 5.6% of all web and API requests failed, peaking at 8.9%, and more than 12,000 organizations saw failed requests. Up to 26% of Git authentication requests failed, and 17% of Actions workflow runs failed with infrastructure errors. In the second outage, 6.2% of web and API requests failed, peaking at 9.3%, and more than 11,000 organizations were affected. Up to 25% of Git authentication requests failed, and 19% of Actions workflow runs failed. No databases experienced data loss, and operations retried after recovery succeeded. GitHub relies on an internal service that keeps track of which database servers should receive new and updated data. While replacing machines that run this service as part of regular maintenance, one of the new machines started with an outdated copy of its records, which still listed database servers we had already retired. When this node came online, it incorrectly modified routing configuration with the outdated records, routing write requests to non-existent database servers. Each outage ended when one of our established machines with correct configuration became the leader of the quorum and sent the correct configuration. After the first outage, we identified the new machine as the source and paused automated maintenance to stop it from being leader again. At 16:52 UTC, an automated maintenance task that did not honor the pause put it back in charge, and the outage happened again. We shut that machine down at 17:02 UTC and have since removed it, and the automation now honors the pause, which was a new regression due to an update of our puppet infrastructure. To prevent recurrence, we are clearing retired database servers from the service's records before any new machine is added, and we are keeping new machines from being elected leader until their records have been checked. We are also changing our systems to reject configuration that point to servers that don't exist and to keep using the last known-good settings instead. Finally, we are giving responders a faster way to take a misbehaving machine out of service, and adding alerts so changes like this are spotted within seconds. We will share a more detailed write-up of these incidents in our monthly availability report, along with the status of the repairs we are making to prevent issues like these in the future.