Ubidots incident
Performance Degradation in Bulk Data Processing
Ubidots experienced a critical incident on February 24, 2024 affecting HTTP Post and TCP and 1 more component, lasting 16d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 26, 2024, 02:22 PM UTC
We are currently investigating a significant performance degradation in our bulk data processing system. This performance bottleneck is impacting the speed of the bulk data endpoint for all users and this incident has the potential to cause significant delays in triggering conditional events and alarms. Our team is currently investigating the root cause and working towards a resolution to restore normal operations as quickly as possible. We have identified that the TTS Plugin is also affected. The reason is due to this plugin relies on the Bulk endpoint to ingest data. We apologize for any inconvenience this may cause and appreciate your patience. Further updates will be provided as we progress in resolving this issue.
- monitoring Feb 26, 2024, 02:28 PM UTC
We are actively working on implementing changes to mitigate the issue.
- monitoring Feb 26, 2024, 02:39 PM UTC
We are still monitoring this latency. There's a substantial number of dots that are currently queued for ingestion. To mitigate the TTS Plugin data ingestion issue, we have deployed a new version that no longer utilizes the affected endpoint.
- monitoring Feb 26, 2024, 07:41 PM UTC
We are continuing to monitor for any further issues.
- resolved Mar 11, 2024, 04:44 PM UTC
This incident has been resolved.