Xink incident
US: Sync service is unavailable. Signatures are not updated (synced) to outlook-addin/client.
Xink experienced a major incident on October 26, 2023 affecting Outlook Add-in, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 25, 2023, 07:31 PM UTC
Our developers are already investigating on this issue
- identified Oct 25, 2023, 10:27 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Oct 25, 2023, 10:38 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 26, 2023, 08:34 AM UTC
This incident has been resolved.
- postmortem Oct 26, 2023, 08:45 AM UTC
# RCA: Microsoft automatic security update of the US cluster triggered before scheduled maintenance. **Summary:** Microsoft automatic security update of the US cluster started before our scheduled maintenance off working hours. Xink sync service failed. US clients were unable to sync their signatures. **Incident Details:** The incident was primarily caused by a failed Xink sync service pod. Our maintenance window is set to weekly, Saturday, 8 hours long. Unexpectedly, Microsoft triggered update on Wednesday during US pick hours. Sync service becomes slow/unavailable, causing signatures not syncing to add-in/client. **Mitigation Steps:** In response to the incident, our team took the following steps to address the issue: 1. Restart failed Kubernetes pod. 2. Disable automatic cluster updates. **Next Steps:** We will disable automatic updates by Microsft and perform it manually according to the current shedule. We will contact Microsft for further explanation.