Proof incident

Notarize application: increased load impacting RON meeting starts

Minor Resolved View vendor source →

Proof experienced a minor incident on September 30, 2021 affecting Proof Consumer (Web) and Notary Platform, lasting 1h 1m. The incident has been resolved; the full update timeline is below.

Started
Sep 30, 2021, 02:47 PM UTC
Resolved
Sep 30, 2021, 03:48 PM UTC
Duration
1h 1m
Detected by Pingoru
Sep 30, 2021, 02:47 PM UTC

Affected components

Proof Consumer (Web)Notary Platform

Update timeline

  1. investigating Sep 30, 2021, 02:47 PM UTC

    We are observing increased error rates related to signers trying to enter a RON meeting. Some notaries are unable to accept meeting requests. Some signers are observing increased wait times. On-call engineering team is investigating the issue.

  2. monitoring Sep 30, 2021, 03:02 PM UTC

    The on-call team applied a short term fix around 10:40 am Eastern time. We are seeing meeting acceptance and the error rates going back to normal. We will continue to monitor the situation closely.

  3. resolved Sep 30, 2021, 03:48 PM UTC

    This issue is resolved. In the process of troubleshooting it, we also identified that our PKI provider experienced an outage which impacted notary service operations. We will be regrouping internally to perform an RCA process and improve our operations going forward.