Fluid Attacks incident

Service availability issues

Major Resolved View vendor source →

Fluid Attacks experienced a major incident on December 4, 2025 affecting Platform, lasting 2h 3m. The incident has been resolved; the full update timeline is below.

Started
Dec 04, 2025, 05:21 PM UTC
Resolved
Dec 04, 2025, 07:24 PM UTC
Duration
2h 3m
Detected by Pingoru
Dec 04, 2025, 05:21 PM UTC

Affected components

Platform

Update timeline

  1. identified Dec 12, 2025, 08:28 PM UTC

    It has been identified that the Platform is experiencing a temporal accessibility degradation.

  2. resolved Dec 12, 2025, 08:28 PM UTC

    The incident has been resolved, and the platform is now operating normally.

  3. postmortem Dec 12, 2025, 08:37 PM UTC

    **Impact** At least one user experienced a brief service interruption. The issue started on UTC-5 25-12-04 14:20 and was proactively discovered 1 minute \(TTD\) later by a staff member who noticed that the platform stopped responding for a few minutes during an update. The problem was resolved in 3 minutes \(TTF\), resulting in a total window of exposure of 4 minutes \(WOE\) [\[1\]](https://gitlab.com/fluidattacks/universe/-/issues/19338). **Cause** While we were improving the machines that run the service, the system replaced the old ones with new ones. However, it turned off all the old machines before the new ones were fully ready, leaving the platform with no way to handle incoming requests for a short period of time [\[2\]](https://gitlab.com/fluidattacks/universe/-/merge_requests/90009). **Solution** A safeguard was added so the system cannot turn off any machine until at least one new one is fully ready. This ensures the service always stays available during updates [\[3\]](https://gitlab.com/fluidattacks/universe/-/merge_requests/90085). **Conclusion** The fix applied ensures that new machines must be ready before old ones are removed. We are also working on performance improvements and will continue reviewing how the system behaves in these situations to see if more protections are needed. **INFRASTRUCTURE\_ERROR < PERFORMANCE\_DEGRADATION**