Datacake experienced a minor incident on May 23, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved May 23, 2024, 06:44 AM UTC
Type: Incident Duration: 1 day, 4 hours and 21 minutes Affected Components: Web Application, Reports, MQTT Server, Outgoing Webhooks May 23, 06:44:19 GMT+0 - Investigating - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly. May 24, 11:05:23 GMT+0 - Resolved - All queues have been processed. We are still investigating the root cause for the unusual buildup and have already taken measures in the form of automated scaling to prevent such situations from happening again in the future. May 23, 13:48:21 GMT+0 - Monitoring - Most queues have been successfully processed. However, MQTT and outgoing Webhooks are still experiencing delays. Our teams are working to resolve these remaining issues as quickly as possible. May 23, 08:12:11 GMT+0 - Identified - We have identified the issue to be related to certain background workers not functioning as expected. We're actively working on implementing improvements to optimize their performance and resolve the issue.