Rev.io incident

Investigating: Sporadic error messages across all web-facing systems.

Minor Resolved View vendor source →

Rev.io experienced a minor incident on July 13, 2020 affecting Rev.io Web and BillCenter.net Web and 1 more component, lasting 2h 17m. The incident has been resolved; the full update timeline is below.

Started
Jul 13, 2020, 04:08 AM UTC
Resolved
Jul 13, 2020, 06:25 AM UTC
Duration
2h 17m
Detected by Pingoru
Jul 13, 2020, 04:08 AM UTC

Affected components

Rev.io WebBillCenter.net WebAgentClick WebREST APISOAP APIDeveloper Documentation (developers.rev.io)

Update timeline

  1. investigating Jul 13, 2020, 04:08 AM UTC

    Sporadic error messages across all web-facing systems have been observed by our monitoring platform. We are currently investigating this issue.

  2. resolved Jul 13, 2020, 06:25 AM UTC

    During routine maintenance, one of the clustered database servers moved to a node that didn't have adequate resources to bring it online. We immediately adjusted both the resources on that cluster node and the node failover priorities. We will continue to review this and perform an overall assessment of our cluster failover configuration. Please reach out to [email protected] if you experience any further issues or have any questions.

  3. postmortem Jul 13, 2020, 06:26 AM UTC

    During routine maintenance, one of the clustered database servers moved to a node that didn't have adequate resources to bring it online. We immediately adjusted both the resources on that cluster node and the node failover priorities. We will continue to review this and perform an overall assessment of our cluster failover configuration.