EcoStruxure IT experienced a major incident on April 17, 2021 affecting EcoStruxure IT Expert and EcoStruxure Asset Advisor and 1 more component, lasting 21h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 17, 2021, 10:08 AM UTC
Our automated monitoring system have detected a possible service issue. Our engineers are currently investigating and we will update the status page shortly.
- identified Apr 17, 2021, 10:24 AM UTC
We are currently experiencing a major outage in our data center. We are still receiving and storing data, but measurements and data processing is delayed. Web, mobile, and desktop applications are degraded or unavailable. Alarm processing is unaffected. No data has been lost.
- identified Apr 17, 2021, 10:43 AM UTC
Our Data Center provider has identified a possible fix for the underlying cause and has engaged additional teams to assist with mitigation steps.
- identified Apr 17, 2021, 10:49 AM UTC
We are observing approximately 30 min delays on measurements. Alarms processing is unaffected.
- identified Apr 17, 2021, 12:53 PM UTC
The underlying data center issue seems to be resolved. But the issue caused instability in some data stores. This is causing the data processing to be delayed. Measurements and other device updates are approximately 1 hour delayed. Alarms processing is unaffected.
- monitoring Apr 17, 2021, 06:24 PM UTC
We have implemented a fix. The performance is improving, but there is still a lot of measurements that need to be processed. Currently, we are observing 5 hours of delay on measurements and other device updates. Alarms are processed without delay. We are monitoring the system closely.
- resolved Apr 18, 2021, 07:18 AM UTC
This incident has been resolved.