Universign incident
October 2023: Universign service degradation on Transaction Service
Universign experienced a minor incident on October 18, 2023 affecting Transaction Service and Timestamp Service and 1 more component, lasting 2h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 18, 2023, 01:46 PM UTC
We are currently experiencing difficulties on our platform. More information will be made available as it is conveyed to us.
- identified Oct 18, 2023, 02:22 PM UTC
We are facing a minor outage. The problem is identified and we are working on a fix right now.
- identified Oct 18, 2023, 02:42 PM UTC
We are continuing to work on a fix for this issue.
- identified Oct 18, 2023, 03:08 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Oct 18, 2023, 03:40 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 18, 2023, 04:12 PM UTC
This incident has been resolved. We are preparing a global report about this incident. We deeply apologize for the inconvenience.
- postmortem Oct 27, 2023, 11:50 AM UTC
We apologize for the inconvenience caused by the incident on the 18/10/2023 between 3:05 PM and 5:08 PM. **Timeline**: 18/10/2023 3:05 PM: Latency alarm on WS service 18/10/2023 3:09 PM: Intermittent alarms on unavailability of WS service **Resolution of the incident:** 18/10/2023 3:09-4:15 PM: Backoffice stopped 18/10/2023 3:09-4:25 PM: Reboot of overloaded servers 18/10/2023 3:20 PM: Server taken out and back into the pool 18/10/2023 4:25 PM: Server taken out and back into the pool 18/10/2023 4:31 PM: IP address blocked **End of incident:** 18/10/2023 5:08 PM: All services are operational 19/10/2023 12:27 PM: IP address unblocked **Identified root cause:** We are investigating high contention at the database level which caused increased latency of queries, going so far as to make the service unavailable. **Preventive actions:** We are investigating high contention at the database level which caused increased latency of queries, going so far as to make the service unavailable. We are investigating changing our database management service. We are investigating integrating a query mitigation service.