AlsoEnergy experienced a major incident on June 2, 2025 affecting Data Ingest, lasting 51d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 02, 2025, 01:27 PM UTC
We are currently investigating this issue.
- monitoring Jun 02, 2025, 05:07 PM UTC
The issue has been identified and resolved. We will continue to monitoring this until all device data has been backfilled.
- monitoring Jun 02, 2025, 05:10 PM UTC
We are continuing to monitor for any further issues.
- investigating Jun 05, 2025, 11:22 PM UTC
We are investigating an issue causing delays. After monitoring, we are looking back into this initial issue.
- identified Jun 10, 2025, 07:44 AM UTC
The issue has been identified and a fix is being implemented.
- investigating Jun 12, 2025, 01:54 PM UTC
We are currently investigating this issue.
- identified Jun 14, 2025, 06:24 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jun 16, 2025, 02:05 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jun 19, 2025, 08:24 PM UTC
We're continuing to monitor system performance and are seeing steady improvements in the sites that are currently backlogged. Thank you for your patience.
- monitoring Jun 20, 2025, 12:49 PM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 21, 2025, 03:07 AM UTC
Scheduled maintenance work resulted in our primary servers reporting severely degraded performance processing incoming data. Upon further investigation, the degradation was ultimately tracked to one of our replicated databases experiencing increased CPU and i/o pressure. Our teams continued the investigation and narrowed the performance issue to a configuration item limiting the memory usage of the replicated server. On 6/13/2025 at 22:00 MST, the configuration was updated, and the replicated database was recycled, allowing the database to return to normal operating conditions. As the system returned to normal processing speeds, it has been continuing to bring the backlogged data up to our servers to be available on our platform. We are aware of sites with remaining backlogged data and are working to provide additional capacity to bring them up to date.
- monitoring Jun 21, 2025, 02:02 PM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 23, 2025, 06:44 AM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 26, 2025, 07:11 PM UTC
Our data operations have returned to normal operating conditions, we are continuing to monitor
- investigating Jun 27, 2025, 01:54 PM UTC
We are currently investigating this issue.
- identified Jun 27, 2025, 02:39 PM UTC
The issue has been identified and a fix is being implemented.
- identified Jun 30, 2025, 01:40 PM UTC
We have identified and addressed the issue affecting our data ingest, we are currently working on increasing the ingest capacity to address data backlog
- monitoring Jul 02, 2025, 02:17 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jul 04, 2025, 10:45 AM UTC
The implemented fix remains in place and is being closely monitored.
- monitoring Jul 08, 2025, 07:26 AM UTC
System stability remains consistent. Monitoring efforts are ongoing to ensure no recurrence of the issue.
- monitoring Jul 11, 2025, 02:42 PM UTC
We’re observing steady system performance with no further issues reported. Monitoring efforts are ongoing to ensure continued stability.
- investigating Jul 21, 2025, 01:43 PM UTC
We are currently investigating this issue.
- investigating Jul 24, 2025, 04:59 PM UTC
We are continuing to investigate this issue.
- resolved Jul 24, 2025, 05:00 PM UTC
This incident has been resolved.