September 2023: Universign major outage on Transaction Service

Update timeline

1. investigating Sep 28, 2023, 09:37 AM UTC

   We are currently experiencing difficulties on our platform. More information will be made available as it is conveyed to us.

2. identified Sep 28, 2023, 09:40 AM UTC

   We are facing a minor outage. The problem is identified and we are working on a fix right now.

3. identified Sep 28, 2023, 10:02 AM UTC

   We are facing a major outage. The problem is identified and we are working on a fix right now.

4. identified Sep 28, 2023, 10:20 AM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Sep 28, 2023, 10:26 AM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Sep 28, 2023, 11:09 AM UTC

   This incident has been resolved. We are preparing a global report about this incident. We deeply apologize for the inconvenience.

7. postmortem Sep 29, 2023, 03:41 PM UTC

   We apologize for the inconvenience caused by the incident between 28/09/2023 11:27 AM and 12:28 PM. Detection time: • 09/28/2023 11:27 a.m.: alert on unavailable services • 09/28/2023 12:21 p.m.: application alerts, degraded service Resolution : • 09/28/2023 11:32 a.m.: restart of impacted network equipment • 09/28/2023 12:21 p.m.: implementation of new network routes • 09/28/2023 12:28 p.m. - 12:41 p.m.: restart of application services Incident end time: • 09/28/2023 12:21 p.m.: U1 services are degraded \(<5% of requests impacted\) • 09/28/2023 12:52 p.m.: all U1 services are operational Cause of the incident: Updating network equipment caused a redundancy fault which is currently being investigated with the manufacturer. Remediation : Ports on other network devices have been reset, returning to the original network routes. An investigation with the equipment manufacturer is underway to understand why the switch was not automatic and transparent. At the same time, we will study the replacement of this network equipment.