Artifik experienced a critical incident on March 6, 2024 affecting Procurement, lasting 1h 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 06, 2024, 08:05 AM UTC
We are currently investigating this issue.
- identified Mar 06, 2024, 09:06 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 06, 2024, 09:06 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 06, 2024, 09:35 AM UTC
This incident has been resolved.
- postmortem Mar 06, 2024, 09:48 AM UTC
The initiating procedure for SMS-based logins was rolled back to fix the problem. Furthermore we’ll look at the routines of updating this, adding it to our automated deploy routines to avoid human failure in the future. We have also set up an alarm for this specific function, so that if the same type of error happens again, we’ll be alerted before any users report the problem to [email protected]. Fallbacks of using Google and password-based login proved useful during this incident. We’ll evaluate if and how we can use the same or similar for any future problems with SMS login.