Prescreen experienced a critical incident on August 20, 2021 affecting onlyfy.io and onlyfy.jobs and 1 more component, lasting 6h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 20, 2021, 12:31 PM UTC
Liebe Prescreen User. Aufgrund von unerwarteten Wartungsarbeiten, ist Prescreen im Moment nicht erreichbar. Wir arbeiten bereits an einer Lösung und melden uns sobald die Seite wieder erreichbar ist. In der Zwischenzeit möchten wir uns für die entstandenen Umstände entschuldigen. Beste Grüße Ihr Prescreen-Team Dear Prescreen User. Due to unexpected maintenance, Prescreen is currently unavailable. We are already working on a solution and will contact you as soon as the site is available again. In the meantime, we apologise for any inconvenience caused. Best regards Your Prescreen Team
- identified Aug 20, 2021, 12:31 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Aug 20, 2021, 02:18 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Aug 20, 2021, 02:20 PM UTC
We are continuing to monitor for any further issues.
- identified Aug 20, 2021, 02:22 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Aug 20, 2021, 05:42 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 20, 2021, 06:37 PM UTC
This incident has been resolved.
- postmortem Aug 24, 2021, 01:52 PM UTC
Dear Prescreen Users, On behalf of the entire Prescreen Team, **I sincerely apologize** to all of you who have been affected by our system outage **on August 20, 2021**, for the inconvenience we have caused. We are aware that Prescreen is an essential part of your processes and that we therefore have a responsibility to you to guarantee the availability and quality of our application. We aspire to be a reliable and transparent partner and would hence like to inform you about what has happened and the measures we have taken: Our distributed file-storage system stopped responding at 1:40 PM that Friday, which is why our automated health checks rightfully set our application offline. Our operations team instantly started investigating and was able to narrow down the cause step by step. Our primary objective was to ensure that all data is safe. For this reason, it unfortunately took some time until our application was able to go online again at 7 PM that day. When restoring files, a temporary error occurred which led to additional issues in the days following the outage: * Some candidates were unable to upload their CV files. * In rare cases, e-mail messages and e-mail applications could not be processed properly until noon of August 24. We triggered a reprocessing of those on Wednesday evening \(August 25\). We have taken the following measures to prevent such incidents in the future: * Review of our storage system together with external experts * Evaluation of alternative storage systems * Introduction of additional monitoring to identify and correct errors even more quickly in the future If you have any further questions or would like a more in-depth exchange with us, please do not hesitate to contact us. We really appreciate working with you and thank you for your trust. Sincerely, Robert Rainer, VP Engineering & Co-Founder and the whole Prescreen Team