CYPHER Learning experienced a major incident on August 13, 2024 affecting CYPHER for business, lasting 1h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 13, 2024, 02:48 PM UTC
Some CYPHER for Business users are experiencing issues in their portal. The team exploring now as a top priority.
- resolved Aug 13, 2024, 04:29 PM UTC
There were a very large number (1m+) orders associated with the "visitor" account that must have accumulated over the past year due to users not fulfilling orders. Today our caching code that caches orders for a user was triggered for the visitor account, resulting on heavy prolonged loading of those 1m accumulated orders. It took at around 45m for us to determine the root cause, since this was a novel edge case that we have never seen before. We then patched our code to ignore these specific calls for the visitor account, and then the site went back to normal again. The longer-term solution which will we will implement shortly is to periodically flush orders that accumulate for the visitor account.