Empist experienced a minor incident on January 19, 2017, lasting 21h 3m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Jan 19, 2017, 04:19 PM UTC
We have detected and investigating some latency in our cloud environment that is impacting performance for some users. We will provide an update as soon as we have identified the cause of the latency.
- identified Jan 19, 2017, 06:18 PM UTC
We've identified the source of the performance degradation to be storage-related and are in the process of working on the SANs in our datacenter to stabilize the environment. We will provide an update within the hour or as soon as the storage issues are addressed and performance has returned to a fully operational status, whichever comes first.
- monitoring Jan 19, 2017, 08:03 PM UTC
We are continuing to work with the manufacturer of the SAN conducting ongoing performance monitoring and testing. However, as of 12:45 pm, we started seeing service times improving and they continue to improve. We will continue monitoring and updating through resolution.
- monitoring Jan 20, 2017, 12:15 AM UTC
While we have seen continued service time improvements over the past few hours that have resulted in a return to full service for most users, there are still users that continue to be impacted. We are aware of the source on our storage devices causing degraded performance for these users and are actively taking steps to resolve these issues. While these efforts will continue until full service is restored, we expect it to take several more hours for a return to full cloud optimization for all users. We will continue to monitor performance analytics during this time and will update accordingly.
- monitoring Jan 20, 2017, 06:58 AM UTC
We are nearly at full cloud optimization across all users. We will continue to monitor overall performance for the next few hours to ensure that full service is restored and that there is no persistent performance degradation for any users. If performance analytics remain consistent, the next update will be that the issue is deemed resolved.
- resolved Jan 20, 2017, 01:23 PM UTC
We monitored system performance throughout the night and this is issue is now set to resolved. All Cloud Services are fully operational across all users and are performing at optimum levels. Upon detecting the source of the degraded performance on the SAN, we worked with the manufacturer until all equipment was successfully replaced. We also made other system adjustments and will reach out to impacted customers directly to further explain some of the enhancements we've made.