Rev.io experienced a major incident on February 26, 2021 affecting Rev.io Web, lasting 3h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 26, 2021, 01:13 PM UTC
We have identified an issue with network traffic, and as a result client sites may not be loading the login page at this time. Our team has identified the problem and are working on implementing a solution. We'll keep this page updated as soon as updates are received.
- identified Feb 26, 2021, 01:33 PM UTC
We are continuing to work on a fix for this issue, and will update this incident accordingly.
- monitoring Feb 26, 2021, 02:45 PM UTC
A fix has been implemented and we are monitoring the results. Site performance should be normalized at this time. Please reach out to the Client Success team with any questions or issues. Thank you for your patience.
- resolved Feb 26, 2021, 04:20 PM UTC
This incident has been resolved. We expect to have a full root cause analysis and post-mortem details added later today. Please reach out with any questions to the Client Success team.
- postmortem Feb 26, 2021, 09:20 PM UTC
One of our datacenters had a hardware failure on a highly-available clustered switch. That did not recover in the appropriate manner and the network links were brought up manually by the infrastructure team. Remediation steps are being undertaken to create a new circuit in order to enable an additional level of failover should a similar issue occur in the future.