Evalink Outage History

No further anomalies have been observed in the evalink API Gateway. The situation is therefore considered resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved - all past and realtime data is now correctly reflected in evalink analytics again.

This incident has been resolved.

This incident has been resolved.

The degraded performance affecting evalink talos in the EU region has been resolved. Our team has confirmed that service performance is operating normally, and this incident is now closed.

This incident has been resolved.

We didn't notice further issues and the system is operating stable since.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

The issue affecting the Alarm Management module in our European region has been fully resolved. All systems are operating normally. A deeper investigation has shown that while alarm transmission remained functional throughout the incident, some alarms may have experienced processing delays during the affected period. No alarms were lost. We have identified that the unexpected downtime was triggered by an infrastructure event during routine operations in the Europe region. Our team is conducting a full root cause analysis. A detailed post-mortem report, including corrective and preventive measures, will be shared in the coming days.

During the deployment of our patch 2.15.1 there was elevated load on the workflow service - which caused longer than usual workflow creation times between 14:35 and 14:43 CET, in some cases having the operation to assign the alarm to a workflow fail. The rest of the system and alarm reception / processing wasn't affected. This was caused by an improvement that we're doing to prevent duplicate alarm assignment in specific timings, depending on how much data some customers have it caused the assignment operation to timeout in the duplicate check.

Final incident summary: A major power blackout in Southwestern Europe and adjacent regions caused an exceptional load increase on the platform. This resulted in slower service response times between 12:35 and 13:55 as the system scaled up to absorb the load. Some less noticeable slowdowns occurred between 13:55 and 14:04 while further stabilization activities were introduced to support the system in handling remaining background transactions. Alarm transmission and processing remained operational throughout and were not seriously affected. All services have been fully stable since, and we continue to monitor the platform closely.

We observed a temporary degradation in the automatic workflows component of the Alarm Management module in the Frankfurt region. The issue occurred for approximately 5 minutes around 22:45 CET and affected a subset of customers, leading to temporary restrictions in the execution of automated workflows. During this period, affected alarms were successfully forwarded to the Alarm Management UI for manual processing. The situation has since stabilized, and the service is operating normally again. Alarm transmission was not affected at any time. We continue to monitor the system closely and will provide further updates if necessary.

A corrupted database index impacted search functionality in our Frankfurt region, leading to increased database load and a temporary degradation in overall system performance. We initially deployed a temporary fix to bypass the corrupted index, restoring search functionality and stabilizing performance. A permanent fix has now been implemented to improve search performance independently of the index. Additionally, the index has been recovered to restore optimal system efficiency. Alarm transmission was not affected at any time. The issue is now fully resolved, and no further impact is expected.

The planned database maintenance in the EU region has been completed, and requests are no longer being delayed. Our team is investigating the cause of the unexpected delays observed during the maintenance.

The evalink platform encountered some slowness between 13:04 and 13:09 CET, affecting only the user interface, backend processing wasn't affected. The immediate measures we've put in place yesterday mitigated the major problems so most users seem to have only noticed the user interface loading slower than usual for a couple minutes. We've identified the component causing those problems and were able to disable it while we work on a full resolution. As part of this incident nxgen genesis platform wasn't able to transmit alarms to evalink between 13:08 and and 13:26, at 13:26 we've resumed alarm transmission from genesis and all alarms that accumulated in this time were transmitted successfully.

Between 13:02 and 13:09 CET our platform was experiencing a high database load causing troubles accessing our services, mostly resulting in longer loading times and some requests failing entirely. No alarms were lost during this incident and everything was processing normally, just slower than usual. This incident was caused by a query causing a high spike in database read load, we've been alerted at 13:04 about this issue by our automated monitoring and managed to quickly bring down the load by implementing database level measures and at 13:09 the system was behaving normally again. We apologize for any inconvenience this may have caused and thank you for your patience and understanding. If you experience any further issues, please don’t hesitate to reach out to our support team.

The issue seems to be fully resolved and didn't occur anymore in the last hours. You can check the twilio incident here for more information: https://status.twilio.com/incidents/vb15s6y1vfs4

This incident has been resolved.

This incident has been resolved.