Keysight Eggplant experienced a minor incident on January 18, 2023 affecting KPI Performance Data Processing, lasting 63d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 18, 2023, 12:13 PM UTC
Since 6th January 2023 an issue has been observed in the RCI KPI Performance Data Processing pipeline. The KPI Performance dashboard remains fully operational, but the latest data is not available for the duration. All other services remain unaffected. The RCI team has identified the cause and is working on a fix. It is expected that full service will be restored in due course, including backfill for the window of unprocessed data.
- identified Jan 23, 2023, 03:44 PM UTC
The RCI team has identified the cause of this outage and is actively working on bringing this data pipeline back online. Our first goal is to have KPI Performance back online - we will communicate an ETA. We then expect to reprocess any missing data for the duration of the outage. We'll be in touch with more information soon.
- identified Jan 27, 2023, 06:57 PM UTC
Today, data was successfully restored for the duration of the outage for a subset of tenants. It was expected that this would lead to a partial recovery in the KPI Performance Pipeline, meaning those tenants were brought up-to-date. However, another entirely unrelated issue was exposed while the pipeline was catching up. The RCI team is continuing to work on a fix for this issue. Our investigations have also concluded that all other components of the RCI v2 platform are fully operational and at no increased risk. We have de-escalated the severity of this incident accordingly.
- identified Jan 31, 2023, 01:08 PM UTC
We are continuing to work on a fix for this issue.
- identified Feb 10, 2023, 12:07 PM UTC
The RCI team continues to work towards a fully operational KPI Performance report for all customers. We are making good progress towards triage and resolution of operational issues, correction of root causes, and re-processing of historic data. We will provide further updates as we hit milestones along the path to resolution.
- identified Mar 06, 2023, 08:10 AM UTC
We are continuing to work on a fix for this issue. This specific combination of root causes has been resolved and tested. We expect to roll this fix out with partial operations restored w/c the 6th of March, with full operations to follow.
- identified Mar 08, 2023, 04:22 PM UTC
The RCI team is pleased to announce that the first phase of corrective actions has been deployed successfully. A subset of tenants now has KPI Performance showing data up to 27th February 2023. Data in the KPI Performance report will now fall behind until the second phase has been deployed. This second phase is expected w/c 13th March. The remainder of the tenants will not see new data in the KPI Performance until phase 3 has been completed. An ETA will be provided in due course. Thank you for your continued patience while the team works to bring this component back online. The other components of the RCI platform are fully operational and are not considered at risk.
- monitoring Mar 15, 2023, 06:55 PM UTC
Today the RCI team successfully deployed phase 2 of corrective actions to the data processing pipeline feeding RCI KPI Performance Data Processing. Accompanied by further reprocessing of historic data for a subset of tenants, this deployment means that the data pipeline has been verified as fully operational. The tenants who had historic data reprocessed in phase one will have the full range of data available. We will follow up with a third phase for tenants who have not yet had their historical data reprocessed. Such tenants will not see any new data until further notice. We thank you for your continued patience in this matter.
- resolved Mar 23, 2023, 11:13 AM UTC
Yesterday the RCI team successfully released a third phase of KPI Performance fixes. This ensures KPI Performance Data Pipeline will resume processing data once enough history has built up after the phase two deployment (ETA May 2023). Tenants who have already caught up are not affected by phase three. Such tenants will continue to operate fully without any further impact expected. Thank you for your patience through this incident.