Datacake experienced a minor incident on May 23, 2024, lasting 1d 4h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating May 23, 2024, 06:44 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..
- identified May 23, 2024, 08:12 AM UTC
We have identified the issue to be related to certain background workers not functioning as expected. We're actively working on implementing improvements to optimize their performance and resolve the issue..
- monitoring May 23, 2024, 01:48 PM UTC
Most queues have been successfully processed. However, MQTT and outgoing Webhooks are still experiencing delays. Our teams are working to resolve these remaining issues as quickly as possible..
- resolved May 24, 2024, 11:05 AM UTC
All queues have been processed. We are still investigating the root cause for the unusual buildup and have already taken measures in the form of automated scaling to prevent such situations from happening again in the future..