# GitHub incident delayed password reset and device verification emails

DevFeed: [GitHub incident delayed password reset and device verification emails](<https://devfeed.tech/articles/we-are-investigating-reports-of-degraded-performance-76243.md>)

Original publisher: [Read original article](<https://www.githubstatus.com/incidents/b6f4j9ljz9r6>)

Published: 2024-04-14T14:21:00Z

Content type: news

Language: en

Sources: [GitHub Status - Incident History](<https://devfeed.tech/sources/github-status-incident-history.md>)

Topics: [Deployment & Scaling](<https://devfeed.tech/topics/deployment-scaling.md>), [Monitoring & Alerting](<https://devfeed.tech/topics/monitoring-alerting.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [dora metrics](<https://devfeed.tech/topics/dora-metrics.md>), [password reset](<https://devfeed.tech/topics/password-reset.md>)

Tags: [2fa](<https://devfeed.tech/tags/2fa.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [delay](<https://devfeed.tech/tags/delay.md>), [detection](<https://devfeed.tech/tags/detection.md>), [email-delivery](<https://devfeed.tech/tags/email-delivery.md>), [incident](<https://devfeed.tech/tags/incident.md>), [password](<https://devfeed.tech/tags/password.md>), [performance](<https://devfeed.tech/tags/performance.md>), [queues](<https://devfeed.tech/tags/queues.md>), [reports](<https://devfeed.tech/tags/reports.md>)

## AI overview

GitHub reports that delayed email delivery affected password resets and device verification for some users from April 11 to April 14. A shared resource pool saw increased usage, while a separate unhealthy job queue blocked the mailer queue. GitHub paused that queue, improved anomaly detection, and added a bypass for time-sensitive emails as a mitigation.

## Source excerpt

Beginning at 17:30 UTC on April 11th and lasting until 20:30 UTC on April 14th, github.com saw significant (up to 2 hours) delays in delivering emails. At 14:21 UTC on April 14th, community reports of this were confirmed and an incident declared. Emails most impacted by the delay were password reset and unrecognized device verification, which contain time-sensitive links or verification codes, and are required to be acted on in order for password resets or unrecognized logins to proceed. Users attempting to reset their password during the incident were unable to complete the reset. Users without two-factor authentication (2FA), signing in on an unrecognized device, were unable to complete device verification. Enterprise Managed Users, users with 2FA, and users on recognized devices or IP addresses were still able to sign in. This impacted 800-1000 user device verifications and 300-400 password resets. The mailer delays were caused by increased usage of a shared resource pool; a separate internal job queue became unhealthy and prevented the mailer queue from being worked on. We have made some immediate improvements to better detect and react to this type of situation again. As a short-term mitigation strategy, we have added a queue-bypass ability for time-sensitive emails, like password reset and unrecognized device verification. We can enable this setting if we observe email delays reoccurring, which will ensure that future incidents do not affect user ability to complete critical login flows. We have paused the unhealthy job queue, to prevent impact to other queues using shared resources. And we have updated our methods of detection for anomalous email delivery, to identify this issue sooner.