Datacake experienced a minor incident on August 30, 2024, lasting 7h 45m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Aug 30, 2024, 06:14 AM UTC
Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly..
- identified Aug 30, 2024, 06:36 AM UTC
We have identified delays in data processing due to an overloaded queue and have already scaled up our processing power for faster performance. Our team is closely monitoring the queue to ensure it is cleared efficiently..
- identified Aug 30, 2024, 09:03 AM UTC
The main affected queue has been successfully processed. However, due to the high volume of data, some secondary queues, such as MQTT and database insertion, are experiencing high load. Data delays are still occurring, and we are actively managing this to minimize impact..
- monitoring Aug 30, 2024, 11:59 AM UTC
We are pleased to report that almost all important queues have caught up and are now processing data in real-time. The MQTT publish queue is still slightly behind, and we are working to resolve this..
- resolved Aug 30, 2024, 01:59 PM UTC
All queues have now finished processing and are operating in real-time. Thank you for your patience..