Routific experienced a major incident on February 9, 2024 affecting Routific API and Routific SaaS product, lasting 7h 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 09, 2024, 02:23 PM UTC
We are currently experiencing some stability issues within our application. We are working to determine exactly what is affected at this time and apologize for any issues this may cause.
- investigating Feb 09, 2024, 03:17 PM UTC
The ongoing issue is currently affecting customer profiles. We are continuing our investigation to resolve this as soon as possible.
- monitoring Feb 09, 2024, 04:34 PM UTC
We have taken some action to resolve this issue. Customer profiles are currently working again at the moment. We are monitoring the situation closely.
- resolved Feb 09, 2024, 09:59 PM UTC
We have isolated the cause of this issue which was caused by a long running task in our backend and resolved it. Customer profiles are again fully functional. Sorry for any inconvenience caused this morning.
- postmortem Feb 15, 2024, 05:27 PM UTC
At approximately 5:00am pst we discovered an issue with one of our services. At this point we weren't sure of the impact yet and but continued to investigate in order to resolve the issue as soon as possible. We later discovered, around 7:00am pst that the issue we were investigating was affecting access to customer profiles. We tried a few actions that temporarily restored access but ultimately we did not know the root cause yet at this point and the feature soon reverted to its degraded status. At 8:am pst we discovered our database was under a high load and determined this is what was causing the disruption to the Customer Profiles feature. Shortly after we reset our database and this resolved the load issue on the database as well as restoring access to customer profiles. Our action items to help prevent such issues in the future was to increase our monitoring on our database load to be informed much sooner if any issues are occurring. We are sorry for the impact this may have had on your business.