Proof experienced a major incident on October 29, 2021 affecting Notarize Consumer App (iOS) and Proof Consumer (Web) and 1 more component, lasting 19h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 29, 2021, 09:05 PM UTC
Notarize has become aware of a problem with the system used to finalize documents on our platform. Notary meetings should continue to function properly, but there will be a delay in receiving finalized documents. We are investigating and will continue to update.
- investigating Oct 29, 2021, 09:32 PM UTC
We have now identified that this issue is preventing notaries from connecting to meetings. We are continuing to investigate.
- investigating Oct 29, 2021, 10:02 PM UTC
We are continuing to investigate the source of this problem. We are also investigating a short term fix that would allow notaries to connect to meetings as a first phase. We'll continue to update status every 30 minutes or if there are any new updates worth reporting.
- identified Oct 29, 2021, 10:08 PM UTC
Notarize has identified this issue as stemming from one of our signing certificate providers. The provider has acknowledged and identified the issue on their end and is prioritizing a fix.
- identified Oct 29, 2021, 10:48 PM UTC
We and our certificate provider are continuing to investigate the issue.
- identified Oct 29, 2021, 11:30 PM UTC
Notarize and our certificate vendor have identified the potential source of the issue as being a loop in the certificate chain. When this validation error occurs, it will prohibit a notary from being able to start meetings. Some notaries continue to have access to the system and are able to complete meetings.
- identified Oct 30, 2021, 12:01 AM UTC
Notarize and our certificate vendor are continuing to investigate the certificate verification issue.
- identified Oct 30, 2021, 12:29 AM UTC
Notarize and our certificate vendor are continuing to investigate the certificate verification issue. A subset of notaries continue to be available to start meetings.
- identified Oct 30, 2021, 01:04 AM UTC
Notarize and our certificate vendor are continuing to investigate the certificate verification issue. A subset of notaries continue to be available to start meetings.
- identified Oct 30, 2021, 01:41 AM UTC
While the investigation continues, a code change is being prepared to help further mitigate the certificate related failures.
- identified Oct 30, 2021, 02:23 AM UTC
The engineering team continues to investigate additional mitigation strategies to allow effected notaries to enter meetings.
- identified Oct 30, 2021, 02:59 AM UTC
While the engineering team continues to investigate additional mitigation strategies to allow impacted notaries to enter meetings, the notary support team is also working with impacted notaries to individually resolve their issues.
- identified Oct 30, 2021, 03:42 AM UTC
Software changes are currently being tested to fix the certificate routing issues that have caused some notaries to be unable to start meetings and document finalization delays.
- identified Oct 30, 2021, 04:37 AM UTC
Software changes have been deployed to production and our initial monitoring of the system show that the document finalization tasks are completing normally.
- identified Oct 30, 2021, 05:06 AM UTC
The system shows both meeting starts and document finalization tasks are operating normally.
- monitoring Oct 30, 2021, 05:16 AM UTC
The system shows both meeting starts and document finalization tasks are operating normally. We continue to monitor the system for any residual effects.
- monitoring Oct 30, 2021, 04:08 PM UTC
We are continuing to monitor for any further issues.
- resolved Oct 30, 2021, 04:10 PM UTC
This issue is now resolved. Notary meetings are occurring and documents are being finalized.