Ocrolus experienced a major incident on December 12, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Dec 13, 2023, 05:08 PM UTC
Our email notification system failed to deliver BOOK_VERIFIED events and password reset emails from approximately 3:30pm ET on December 12th to 11:30 AM ET on December 13th. We are still investigating the root cause of this issue.
- postmortem Dec 14, 2023, 05:24 PM UTC
Our email notifications rely on a distributed event streaming system and an email events consumer. This incident was caused by a a failure of that system and a software bug that prevented the event consumer from restarting. We have deployed a fix for the software bug and are adding additional monitoring for our event streaming system.