Datacake experienced a minor incident on August 30, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 30, 2024, 06:14 AM UTC
Type: Incident Duration: 7 hours and 45 minutes Affected Components: Web Application, Reports, MQTT Server, Outgoing Webhooks Aug 30, 06:14:27 GMT+0 - Investigating - Our system is experiencing processing delays. No data has been lost, and the system should be caught up shortly. Aug 30, 06:36:15 GMT+0 - Identified - We have identified delays in data processing due to an overloaded queue and have already scaled up our processing power for faster performance. Our team is closely monitoring the queue to ensure it is cleared efficiently. Aug 30, 13:59:49 GMT+0 - Resolved - All queues have now finished processing and are operating in real-time. Thank you for your patience. Aug 30, 09:03:05 GMT+0 - Identified - The main affected queue has been successfully processed. However, due to the high volume of data, some secondary queues, such as MQTT and database insertion, are experiencing high load. Data delays are still occurring, and we are actively managing this to minimize impact. Aug 30, 11:59:49 GMT+0 - Monitoring - We are pleased to report that almost all important queues have caught up and are now processing data in real-time. The MQTT publish queue is still slightly behind, and we are working to resolve this.