DoControl experienced a major incident on April 1, 2021 affecting Graphql Api, lasting 3h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 01, 2021, 08:09 AM UTC
We spotted an issue while fetching user/assets data from our DB. we are restoring the data from backup and updated in the coming hour.
- investigating Apr 01, 2021, 08:09 AM UTC
We are continuing to investigate this issue.
- investigating Apr 01, 2021, 08:25 AM UTC
We restore the data from back up, the cluster is running on read-only mode till full recovery.
- identified Apr 01, 2021, 08:25 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 01, 2021, 09:02 AM UTC
after restoring from the backup we are monitoring to see if there are more issues.
- monitoring Apr 01, 2021, 09:41 AM UTC
We identified that slack collaborators are missing, we are working on restoring those and resolve the incident.
- monitoring Apr 01, 2021, 09:58 AM UTC
the data is now available and we back online. still monitoring and validating all API's are working correctly.
- monitoring Apr 01, 2021, 10:07 AM UTC
We are continuing to monitor for any further issues.
- resolved Apr 01, 2021, 11:55 AM UTC
This incident has been resolved.