Rev.io experienced a major incident on July 14, 2020 affecting Rev.io Web and BillCenter.net Web and 1 more component, lasting 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 14, 2020, 05:11 AM UTC
A systems outage has occurred. Engineers are aware of the problem and have implemented a fix, it should be deployed shortly.
- identified Jul 14, 2020, 05:16 AM UTC
We are continuing to work on a fix for this issue.
- resolved Jul 14, 2020, 05:57 AM UTC
A fix has been implemented successfully. A full postmortem will be posted after the start of business hours later today.
- postmortem Jul 21, 2020, 10:40 PM UTC
At 01:05 AM EDT on July 14th 2020 a drive image in our VM cluster encountered an error which required technician intervention to restore the drive image to an operable state. Once the drive was restored the majority of the effected services came online almost immediately. A database shared by many of our services did not recover as quickly and this caused downtime for some of our services. The effected database recovered after 1:29 of downtime.