SchemeServe experienced a major incident on March 31, 2025 affecting 🎩 SchemeServe, lasting 1h 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 31, 2025, 07:09 AM UTC
SchemeServe is currently experiencing issues with multi-factor authentication. the issue is under invetigation.
- resolved Mar 31, 2025, 08:37 AM UTC
The incident has been resolved, and MFA is now working correctly again. A post-mortem will be following shortly
- postmortem Mar 31, 2025, 12:47 PM UTC
**Root Cause** SchemeServe’s authentication system relies on short-lived access tokens known as JWTs \(JSON Web Tokens\), each with a lifespan of 15 minutes before requiring renewal. Historically, SchemeServe has used **BST \(British Summer Time\)** across its codebase and storage layers. As part of an ongoing effort to align with international standards, newer services have been transitioned to operate in **UTC**. However, the JWT generation process had not yet been updated and was still using BST. As a result, when the UK clocks moved forward at **01:00 on Sunday, 30th March**, newly generated tokens contained timestamps based on BST. These tokens appeared to be issued in the future relative to the UTC-based verification process, placing them immediately outside the valid lifetime window. Consequently, the tokens were rejected and users were unable to authenticate. **Impact** This issue prevented users from authenticating into **SchemeServe**, effectively blocking access to the platform. **Detection** Unfortunately, this event did not trigger any health alerts and as such, the issue was not detected in real-time. We became aware of the problem at **07:50 on Monday, 31st March**, following user reports. Further investigation quickly identified the root cause. The incident led to prolonged authentication failures from **01:00 Sunday** until resolution. **Resolution** The JWT generation process was updated to explicitly use **UTC**, ensuring that newly issued tokens fall within the expected validity window. This change was deployed, and the issue was fully resolved by **09:34 on Monday, 31st March**. We sincerely regret the disruption caused by this incident and appreciate your understanding as we continue to improve the resilience and reliability of the SchemeServe platform.