Datacake experienced a minor incident on March 14, 2024, lasting 1d 2h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Mar 14, 2024, 11:38 AM UTC
Our system is experiencing processing delays with incoming data. We are currently investigating the cause of this delay..
- investigating Mar 14, 2024, 11:44 AM UTC
We are currently investigating this incident..
- identified Mar 14, 2024, 12:03 PM UTC
We've identified the root cause of the recent issue as an unexpected high load on one of our databases. We've enhanced the resources allocated to this database and are currently monitoring its performance closely. Please note that due to a pending backlog, temporary data gaps may be evident in the chart visualizations. However, we're working to ensure data integrity as soon as possible..
- identified Mar 14, 2024, 12:48 PM UTC
We are still in the process of examining an extended measurement queue. Please note that data transmission still encounters delays..
- identified Mar 14, 2024, 01:24 PM UTC
We've identified the initial root cause of the delays as a few long-running migration queries. We are actively monitoring the situation and adjusting resources as necessary to expedite the process..
- monitoring Mar 14, 2024, 03:16 PM UTC
The queries in question have now stabilized, resulting in a consistent reduction of the queue. We'll continue to provide updates on the situation in this space..
- resolved Mar 14, 2024, 10:02 PM UTC
All queues have now been cleared with no data loss. We apologize for these delays and appreciate your understanding. In dedication to clarity and process improvement, a comprehensive follow-up about the incident, explaining its causes and the measures taken to prevent recurrence, will be provided shortly..
- resolved Mar 15, 2024, 02:24 PM UTC
The full post mortem can be found on our Engineering Blog: .