Get email notifications whenever GitHub creates, updates or resolves an incident.
Get text message notifications whenever GitHub creates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge our Privacy Policy. In addition, you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Get webhook notifications whenever GitHub creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
Git Operations
?
Operational
API Requests
?
Operational
Webhooks
?
Operational
Visit www.githubstatus.com for more information
Operational
Issues
?
Operational
Pull Requests
?
Operational
Actions
?
Operational
Packages
?
Operational
Pages
?
Operational
Codespaces
?
Operational
Copilot
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jul 27, 2024
No incidents reported today.
Jul 26, 2024
No incidents reported.
Jul 25, 2024
Resolved -
This incident has been resolved.
Jul 25, 21:05 UTC
Jul 25, 21:05 UTC
Investigating -
We are currently investigating this issue.
Jul 25, 21:04 UTC
Jul 25, 21:04 UTC
Resolved -
On July 25th, 2024, between 15:30 and 19:10 UTC, the Audit Log service experienced degraded write performance. During this period, Audit Log reads remained unaffected, but customers would have encountered delays in the availability of their current audit log data. There was no data loss as a result of this incident.
The issue was isolated to a single partition within the Audit Log datastore. Upon restarting the primary partition, we observed an immediate recovery and a subsequent increase in successful writes. The backlog of log messages was fully processed by approximately 00:40 UTC on July 26th.
We are working with our datastore team to ensure mitigation is in place to prevent future impact. Additionally, we will investigate whether there are any actions we can take on our end to reduce the impact and time to mitigate in the future.
Jul 25, 19:20 UTC
The issue was isolated to a single partition within the Audit Log datastore. Upon restarting the primary partition, we observed an immediate recovery and a subsequent increase in successful writes. The backlog of log messages was fully processed by approximately 00:40 UTC on July 26th.
We are working with our datastore team to ensure mitigation is in place to prevent future impact. Additionally, we will investigate whether there are any actions we can take on our end to reduce the impact and time to mitigate in the future.
Jul 25, 19:20 UTC
Update -
We have applied a fix and are seeing recovery. (Point of clarification: Impact was constrained to Audit Log Events, not all categories of events.)
Jul 25, 19:16 UTC
Jul 25, 19:16 UTC
Investigating -
We are currently investigating this issue.
Jul 25, 18:44 UTC
Jul 25, 18:44 UTC
Jul 24, 2024
No incidents reported.
Jul 23, 2024
Resolved -
This incident has been resolved.
Jul 23, 22:38 UTC
Jul 23, 22:38 UTC
Update -
We have mitigated the issue with Copilot Chat returning failures in some regions. Functionality has recovered for all Copilot Chat users.
Jul 23, 22:25 UTC
Jul 23, 22:25 UTC
Update -
We are seeing failures for the Copilot Chat for users in some regions. We are seeing about 20% of Copilot Chat requests fail.
Jul 23, 21:52 UTC
Jul 23, 21:52 UTC
Investigating -
We are currently investigating this issue.
Jul 23, 21:40 UTC
Jul 23, 21:40 UTC
Jul 22, 2024
No incidents reported.
Jul 21, 2024
No incidents reported.
Jul 20, 2024
No incidents reported.
Jul 19, 2024
Resolved -
This incident has been resolved.
Jul 19, 04:47 UTC
Jul 19, 04:47 UTC
Update -
Codespaces is still recovering fully, but we can see the issue is trending positive. Please stop and start your Codespace, if impacted: https://docs.github.com/en/codespaces/developing-in-a-codespace/stopping-and-starting-a-codespace?tool=webui
Jul 19, 03:54 UTC
Jul 19, 03:54 UTC
Update -
We are still investigating issues with Codespaces. Some users may not be able to connect to their Codespaces at this time. We will update you on mitigation progress.
Jul 19, 03:17 UTC
Jul 19, 03:17 UTC
Update -
We are investigating issues with Codespaces. Some users may not be able to connect to their Codespaces at this time. We will update you on mitigation progress.
Jul 19, 02:43 UTC
Jul 19, 02:43 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Jul 19, 02:10 UTC
Jul 19, 02:10 UTC
Resolved -
Beginning on July 18, 2024 at 22:38 UTC, network issues within an upstream provider led to degraded experiences across Actions, Copilot, and Pages services.
Up to 50% of Actions workflow jobs were stuck in the queuing state, including Pages deployments. Users were also not able to enable Actions or register self-hosted runners. This was caused by an unreachable backend resource in the Central US region. That resource is configured for geo-replication, but the replication configuration prevented resiliency when one region was unavailable. Updating the replication configuration mitigated the impact by allowing successful requests while one region was unavailable. By July 19 00:12 UTC, users saw some improvement in Actions jobs and full recovery of Pages. Standard hosted runners and self-hosted Actions workflows were healthy by 2:10 UTC and large hosted runners fully recovered at 2:38.
Copilot requests were also impacted with up to 2% of Copilot Chat requests and 0.5% of Copilot Completions requests resulting in errors. Chat requests were routed to other regions after 20 minutes while Completions requests took 45 minutes to reroute.
We have identified improvements to detection to reduce the time to engage all impacted on-call teams and improvements to our replication configuration and failover workflows to be more resilient to unhealthy dependencies and reduce our time to failover and mitigate customer impact.
Jul 19, 02:38 UTC
Up to 50% of Actions workflow jobs were stuck in the queuing state, including Pages deployments. Users were also not able to enable Actions or register self-hosted runners. This was caused by an unreachable backend resource in the Central US region. That resource is configured for geo-replication, but the replication configuration prevented resiliency when one region was unavailable. Updating the replication configuration mitigated the impact by allowing successful requests while one region was unavailable. By July 19 00:12 UTC, users saw some improvement in Actions jobs and full recovery of Pages. Standard hosted runners and self-hosted Actions workflows were healthy by 2:10 UTC and large hosted runners fully recovered at 2:38.
Copilot requests were also impacted with up to 2% of Copilot Chat requests and 0.5% of Copilot Completions requests resulting in errors. Chat requests were routed to other regions after 20 minutes while Completions requests took 45 minutes to reroute.
We have identified improvements to detection to reduce the time to engage all impacted on-call teams and improvements to our replication configuration and failover workflows to be more resilient to unhealthy dependencies and reduce our time to failover and mitigate customer impact.
Jul 19, 02:38 UTC
Update -
Actions is operating normally.
Jul 19, 02:38 UTC
Jul 19, 02:38 UTC
Update -
We have continued to apply mitigations to work around the outage. Customers may still experience run start delays for larger runners.
Jul 19, 02:25 UTC
Jul 19, 02:25 UTC
Update -
We've applied a mitigation to work around the outage. Customers may still experience run start delays.
Jul 19, 01:50 UTC
Jul 19, 01:50 UTC
Update -
We are making progress failing over to a different region to mitigate an outage.
Jul 19, 01:04 UTC
Jul 19, 01:04 UTC
Update -
We continue to mitigate an outage by failing over to a different region.
Jul 19, 00:30 UTC
Jul 19, 00:30 UTC
Update -
Pages is operating normally.
Jul 19, 00:24 UTC
Jul 19, 00:24 UTC
Update -
We are working to mitigate an outage by failing over to a different region.
Jul 18, 23:57 UTC
Jul 18, 23:57 UTC
Update -
Pages is experiencing degraded performance. We are continuing to investigate.
Jul 18, 23:23 UTC
Jul 18, 23:23 UTC
Update -
Some actions customers may experience delays or failures in their runs. We continuing to investigate.
Jul 18, 23:22 UTC
Jul 18, 23:22 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Jul 18, 22:47 UTC
Jul 18, 22:47 UTC
Jul 18, 2024
Jul 17, 2024
Resolved -
This incident has been resolved.
Jul 17, 18:13 UTC
Jul 17, 18:13 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Jul 17, 17:56 UTC
Jul 17, 17:56 UTC
Resolved -
On July 17, 2024, between 16:15:31 UTC and 17:06:53 UTC, various GitHub services were degraded including Login, the GraphQL API, Issues, Pages and Packages. On average, the error rate was 0.3% for requests to github.com and the API, and 3.0% of requests for Packages. This incident was triggered by two unrelated events:
- A planned testing event of an internal feature caused heavy loads on our databases, disrupting services across GitHub.
- A network configuration change deployed to support capacity expansion in a GitHub data center.
We partially resolved the incident by aborting the testing event at 16:17 UTC and fully resolved the incident by rolling back the network configuration changes at 16:49 UTC.
We have paused all planned capacity expansion activity within GitHub data centers until we have stabilized the root cause of this incident. In addition, we are reexamining our load testing practices so they can be done in a safer environment and making architectural changes to the feature that caused issues.
Jul 17, 17:06 UTC
- A planned testing event of an internal feature caused heavy loads on our databases, disrupting services across GitHub.
- A network configuration change deployed to support capacity expansion in a GitHub data center.
We partially resolved the incident by aborting the testing event at 16:17 UTC and fully resolved the incident by rolling back the network configuration changes at 16:49 UTC.
We have paused all planned capacity expansion activity within GitHub data centers until we have stabilized the root cause of this incident. In addition, we are reexamining our load testing practices so they can be done in a safer environment and making architectural changes to the feature that caused issues.
Jul 17, 17:06 UTC
Update -
Git Operations is operating normally.
Jul 17, 17:06 UTC
Jul 17, 17:06 UTC
Update -
Pages is operating normally.
Jul 17, 17:06 UTC
Jul 17, 17:06 UTC
Update -
Packages is operating normally.
Jul 17, 17:06 UTC
Jul 17, 17:06 UTC
Update -
We're continuing to investigate reports of issues with multiple services. We will continue to keep users updated on progress.
Jul 17, 16:47 UTC
Jul 17, 16:47 UTC
Update -
Packages is experiencing degraded performance. We are continuing to investigate.
Jul 17, 16:47 UTC
Jul 17, 16:47 UTC
Update -
Issues is operating normally.
Jul 17, 16:46 UTC
Jul 17, 16:46 UTC
Update -
Pages is experiencing degraded performance. We are continuing to investigate.
Jul 17, 16:30 UTC
Jul 17, 16:30 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Jul 17, 16:28 UTC
Jul 17, 16:28 UTC
Update -
We are investigating reports of issues with service(s): Git, Actions, Rules and Permissions, SSH authentication, and Authorization.. We will continue to keep users updated on progress towards mitigation.
Jul 17, 16:24 UTC
Jul 17, 16:24 UTC
Investigating -
We are investigating reports of degraded performance for Git Operations
Jul 17, 16:21 UTC
Jul 17, 16:21 UTC
Jul 16, 2024
Resolved -
This incident has been resolved.
Jul 16, 03:07 UTC
Jul 16, 03:07 UTC
Update -
Copilot chat is error rates are improving and we are continuing to monitor system health.
Jul 16, 02:56 UTC
Jul 16, 02:56 UTC
Update -
Copilot chat is experiencing elevated error rates. We have identified the root cause and are working on remediation.
Jul 16, 02:10 UTC
Jul 16, 02:10 UTC
Update -
Copilot chat is experiencing elevated error rates and we are currently investigating the issue.
Jul 16, 01:35 UTC
Jul 16, 01:35 UTC
Investigating -
We are currently investigating this issue.
Jul 16, 00:53 UTC
Jul 16, 00:53 UTC
Jul 15, 2024
No incidents reported.
Jul 14, 2024
No incidents reported.
Jul 13, 2024
Resolved -
On July 13, 2024 between 00:01 and 19:27 UTC the Copilot service was degraded. During this time period, Copilot code completions error rate peaked at 1.16% and Copilot Chat error rate peaked at 63%. Between 01:00 and 02:00 UTC we were able to reroute traffic for Chat to bring error rates below 6%. During the time of impact customers would have seen delayed responses, errors, or timeouts during requests. GitHub code scanning autofix jobs were also delayed during this incident.
A resource cleanup job was scheduled by Azure OpenAI (AOAI) service early July 13th targeting a resource group thought to only contain unused resources. This resource group unintentionally contained critical, still in use, resources that were then removed. The cleanup job was halted before removing all resources in the resource group. Enough resources remained that GitHub was able to mitigate while resources were reconstructed.
We are working with AOAI to ensure mitigation is in place to prevent future impact. In addition, we will improve traffic rerouting processes to reduce time to mitigate in the future.
Jul 13, 19:27 UTC
A resource cleanup job was scheduled by Azure OpenAI (AOAI) service early July 13th targeting a resource group thought to only contain unused resources. This resource group unintentionally contained critical, still in use, resources that were then removed. The cleanup job was halted before removing all resources in the resource group. Enough resources remained that GitHub was able to mitigate while resources were reconstructed.
We are working with AOAI to ensure mitigation is in place to prevent future impact. In addition, we will improve traffic rerouting processes to reduce time to mitigate in the future.
Jul 13, 19:27 UTC
Update -
Copilot is operating normally.
Jul 13, 19:26 UTC
Jul 13, 19:26 UTC
Update -
Our upstream provider continues to recover and we expect services to return to normal as more progress is made. We will provide another update by 20:00 UTC.
Jul 13, 18:01 UTC
Jul 13, 18:01 UTC
Update -
Our upstream provider is making good progress recovering and we are validating that services are nearing normal operations. We will provide another update by 18:00 UTC.
Jul 13, 16:09 UTC
Jul 13, 16:09 UTC
Update -
Our upstream provider is gradually recovering the service. We will provide another update at 23:00 UTC.
Jul 13, 11:18 UTC
Jul 13, 11:18 UTC
Update -
We are continuing to wait on our upstream provider to see full recovery. We will provide another update at 11:00 UTC
Jul 13, 03:50 UTC
Jul 13, 03:50 UTC
Update -
The error rate for Copilot chat requests remains steady at less than 10%. We are continuing to investigate with our upstream provider.
Jul 13, 03:20 UTC
Jul 13, 03:20 UTC
Update -
Copilot is experiencing degraded performance. We are continuing to investigate.
Jul 13, 02:20 UTC
Jul 13, 02:20 UTC
Update -
We have applied several mitigations to Copilot chat, reducing errors to less than 10% of all chat requests. We are continuing to investigate the issue with our upstream provider.
Jul 13, 02:19 UTC
Jul 13, 02:19 UTC
Update -
Copilot chat is experiencing degraded performance, impacting up to 60% of all chat requests. We are continuing to investigate the issue with our upstream provider.
Jul 13, 01:32 UTC
Jul 13, 01:32 UTC
Update -
Copilot chat is currently experiencing degraded performance, impacting up to 60% of all chat requests. We are investigating the issue.
Jul 13, 00:49 UTC
Jul 13, 00:49 UTC
Update -
Copilot is experiencing degraded availability. We are continuing to investigate.
Jul 13, 00:29 UTC
Jul 13, 00:29 UTC
Update -
Copilot API chat experiencing significant failures to backend services
Jul 13, 00:18 UTC
Jul 13, 00:18 UTC
Investigating -
We are investigating reports of degraded performance for Copilot
Jul 13, 00:18 UTC
Jul 13, 00:18 UTC
