Telerivet experienced a critical incident on May 22, 2021 affecting Web Dashboard and REST API, lasting 14h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 22, 2021, 01:41 PM UTC
Telerivet is currently investigating an issue with database connectivity.
- investigating May 22, 2021, 01:41 PM UTC
We are continuing to investigate this issue.
- identified May 22, 2021, 01:56 PM UTC
The issue has been identified as a particular query causing Telerivet's database server to crash. Telerivet has disabled scheduled message functionality to avoid this issue and is continuing to investigate the issue.
- identified May 22, 2021, 02:32 PM UTC
We are continuing to work on a fix for this issue.
- monitoring May 22, 2021, 02:39 PM UTC
The issue was caused by a corrupted database table storing scheduled messages and other scheduled events, which caused the database server to crash when triggering scheduled events. The corrupted database table has been repaired, and scheduled messages are being sent now. All Telerivet functionality should be working normally.
- resolved May 23, 2021, 03:47 AM UTC
Telerivet has not detected any further issues with database connectivity or data integrity, and has verified that no scheduled messages or other data were lost due to this issue.