Fluid Attacks Outage History

Started Nov 03, 2026, 10:11 PM UTC· resolved Nov 04, 2026, 10:43 PM UTC

1. identified Nov 03, 2026, 10:11 PM UTC

   Since October 29 at 18:57 GMT-05:00, repositories connected through Bitbucket OAuth credentials cannot be cloned, and their cloning status shows “Invalid credentials, access could not be granted prior to the cloning task”. Adding or re-authorizing Bitbucket OAuth credentials from the Credentials section of the organization also fails, with the Bitbucket error “Invalid client_id”. Repositories that use HTTPS or SSH credentials, or other providers (GitHub, GitLab, Azure DevOps), are not affected, and no data was lost or exposed. Our team is working on a fix.

2. identified Nov 04, 2026, 03:40 PM UTC

   We are continuing to work on a fix for this issue

3. monitoring Nov 04, 2026, 10:39 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 04, 2026, 10:43 PM UTC

   This incident has been resolved.

5. postmortem Nov 05, 2026, 04:26 AM UTC

   #### Impact Some users were unable to clone repositories connected to Bitbucket through OAuth, and could not connect or reconnect Bitbucket through OAuth. The issue started on UTC-5 26-10-29 18:57 and was reactively discovered 2.9 days \(TTD\) later by a staff member who reported, on behalf of a client, that one of its groups had lost access to its Bitbucket repositories. The problem was resolved in 1 day \(TTF\), resulting in a total window of exposure of 3.9 days \(WOE\). * **Cloning**: repositories connected through Bitbucket OAuth were marked as failed, so they did not receive new code for analysis, and their pending reattack requests were put on hold. * **Connections**: users who tried to add or reconnect a Bitbucket OAuth credential received an error from Bitbucket instead of the authorization screen. New connections had probably been failing since October 22. * **Scope**: signing in with Bitbucket, repositories from other providers, and Bitbucket repositories connected with other credential types kept working normally. #### Cause Our platform accesses Bitbucket repositories through an OAuth app, which must be associated with a Bitbucket account. The account that held this app became inactive, and as a result Bitbucket disabled the app and the access it had granted. This change occurred on Bitbucket’s side, with no changes to our platform. Our platform interpreted the resulting errors as a temporary Bitbucket issue, and we did not have an alert for failures affecting all Bitbucket connections at once, so the problem was not detected until it was reported. #### Solution We registered a new Bitbucket OAuth app under a company-managed account, with the same read-only access as before, and switched the platform to use it[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/121323) New Bitbucket OAuth connections now work as expected. Since access granted through the previous app cannot be transferred, affected credentials need to be reconnected from the organization’s Credentials section; our team is reaching out to each affected organization to help with this. We will also document a rotation procedure for this app, add an alert for widespread Bitbucket connection failures, improve how the platform handles this type of error, and use a separate app for our development environment. #### Conclusion This incident showed that a key integration depended on an account that was not centrally managed, and that our monitoring did not distinguish a widespread failure from a temporary provider issue. Managing integration apps under company accounts with documented ownership and rotation steps, and monitoring connection failures across all credentials, would have prevented this outage and allowed us to detect it sooner. **ROTATION\_FAILURE < INCOMPLETE\_PERSPECTIVE < MISSING\_ALERT**

Read the full incident report →

Started Oct 26, 2026, 11:01 PM UTC· resolved Oct 27, 2026, 01:29 AM UTC

1. identified Oct 26, 2026, 11:01 PM UTC

   Since 11:35 GMT-05:00, several AI-powered services on our platform are unavailable: the AI agent chat fails with "Failed to connect to chat service", connections to the Fluid Attacks MCP server (https://app.fluidattacks.com/mcp/messages/) from MCP clients such as Claude Desktop, Claude Code and other AI assistants fail with the error "Unexpected content type: text/html", and pull request reviews from the PR Assistant may not complete.

2. resolved Oct 27, 2026, 01:29 AM UTC

   This incident has been resolved.

3. postmortem Oct 27, 2026, 08:55 PM UTC

   #### Impact Some users experienced an outage of the AI-powered services on our platform: the AI agent chat failed with "Failed to connect to chat service", sessions from external MCP clients could not start, and some pull request reviews from the PR Assistant did not finish processing. The issue started on UTC-5 26-10-26 11:35 and was reactively discovered 5.8 hours \(TTD\) later by a staff member who noticed that the AI agent had stopped responding. The problem was resolved in 1.2 hours \(TTF\), resulting in a total window of exposure of 7 hours \(WOE\). * **AI Agent**: users who opened the AI agent on the platform did not get an answer. * **MCP server**: connections from MCP clients such as Claude Desktop, Claude Code and IDEs failed with "Unexpected content type: text/html". * **PR Assistant**: some pull request review runs never finished processing. * **Scope**: the API, vulnerability management, reports and scans kept working normally, and no data was lost or exposed. #### Cause The services inside our infrastructure locate each other through an internal name resolution service. A change that enabled a new network security layer on some groups of servers modified how name resolution works on those servers[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/118341) On those servers the internal name resolution service ended up forwarding requests to itself, detected the loop, and stopped. Nothing restricted this service to the servers where it works correctly, and since servers are replaced every day, one of its two instances landed on an affected server on October 19 and stopped working. The service kept running on a single instance for a week without any alert. On October 26, an unrelated infrastructure change automatically updated this service to a newer version[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/119719) The restart moved the last healthy instance onto an affected server, leaving no instances available, so every AI service that depends on internal name resolution stopped responding. The new version itself was not the cause; both versions behave the same way on the affected servers. #### Solution The service was restored by restricting the internal name resolution service to servers without the new network layer, after which both instances started correctly. The permanent fix keeps this service on a dedicated group of base servers, the same placement already used by other core components, and raises that group to a minimum of two servers[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/120043) A complementary change fixes the versions of the base components of our infrastructure, so updates are deliberate and reviewed instead of a side effect of unrelated changes[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/120037) The team will also add an alert on the availability of this service. #### Conclusion This incident showed that the change to the network configuration of our servers was validated for the services it targeted but not for the base components of our infrastructure that could run on those servers, so the defect stayed hidden until the daily server rotation exposed it. Without an alert on the availability of name resolution and with component versions not fixed, a routine infrastructure update turned a silent degradation into an outage. Placing base components on dedicated servers, fixing their versions and monitoring their availability would have prevented any user from being affected. **INFRASTRUCTURE\_ERROR < MISSING\_ALERT < INCOMPLETE\_PERSPECTIVE**

Read the full incident report →

Started Oct 01, 2026, 05:25 AM UTC· resolved Oct 01, 2026, 03:58 PM UTC

1. identified Oct 01, 2026, 05:25 AM UTC

   An issue was found in PLATFORM. Click for details: https://availability.fluidattacks.tech

2. resolved Oct 01, 2026, 03:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 01, 2026, 05:10 AM UTC· resolved Oct 01, 2026, 04:01 PM UTC

1. identified Oct 01, 2026, 05:10 AM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.tech

2. resolved Oct 01, 2026, 04:01 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 01, 2026, 05:08 AM UTC· resolved Oct 01, 2026, 04:01 PM UTC

1. identified Oct 01, 2026, 05:08 AM UTC

   An issue was found in AGENT. Click for details: https://availability.fluidattacks.tech

2. resolved Oct 01, 2026, 04:01 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 15, 2026, 03:49 AM UTC· resolved Sep 15, 2026, 05:03 AM UTC

1. identified Sep 15, 2026, 04:19 AM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.tech

Read the full incident report →

Started Sep 11, 2026, 11:52 PM UTC· resolved Sep 12, 2026, 12:24 AM UTC

1. identified Sep 11, 2026, 11:52 PM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.tech

Read the full incident report →

Started Sep 11, 2026, 11:47 PM UTC· resolved Sep 14, 2026, 02:59 PM UTC

1. identified Sep 11, 2026, 11:47 PM UTC

   An issue was found in AGENT. Click for details: https://availability.fluidattacks.tech

2. resolved Sep 14, 2026, 02:59 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 10, 2026, 11:12 PM UTC· resolved Sep 11, 2026, 02:47 AM UTC

1. identified Sep 10, 2026, 11:12 PM UTC

   An issue was found in AGENT. Click for details: https://availability.fluidattacks.tech

2. resolved Sep 11, 2026, 02:47 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 10, 2026, 04:23 AM UTC· resolved Sep 10, 2026, 03:21 PM UTC

1. identified Sep 10, 2026, 04:23 AM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.tech

2. resolved Sep 10, 2026, 03:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 09, 2026, 01:12 PM UTC· resolved Sep 09, 2026, 04:06 PM UTC

1. identified Sep 09, 2026, 01:12 PM UTC

   An issue was found in AGENT. Click for details: https://availability.fluidattacks.tech

2. resolved Sep 09, 2026, 04:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 02, 2026, 08:16 PM UTC· resolved Sep 02, 2026, 08:16 PM UTC

1. identified Sep 02, 2026, 05:55 AM UTC

   An issue was found in DATABASE. Click for details: https://availability.fluidattacks.com

2. resolved Sep 02, 2026, 08:16 PM UTC

   This confirms the incident reported by Checkly for the DATABASE check (Web Group, production tag) was a false alert. The cause was a change to the page title, which the e2e test compared against an exact match ("Fluid Attacks Database"). The database remained available throughout, with no real impact to the service.

Read the full incident report →

Started Aug 25, 2026, 06:42 AM UTC· resolved Aug 25, 2026, 03:02 PM UTC

1. identified Aug 25, 2026, 06:42 AM UTC

   An issue was found in AGENT. Click for details: https://availability.fluidattacks.com

2. resolved Aug 25, 2026, 03:02 PM UTC

   This incident has been resolved. Between 06:52 and 07:12 UTC-5, monitoring detected a single degraded request out of 118 total checks (117 passing, 0 failing). Average response time remained stable at 702 ms, and overall availability held at 99%. The degradation was isolated to one query and did not represent a systemic issue or service disruption. No further degraded or failing requests have been observed since, and the AGENT is operating normally.

Read the full incident report →

Started Aug 25, 2026, 06:42 AM UTC· resolved Aug 25, 2026, 03:04 PM UTC

1. identified Aug 25, 2026, 06:42 AM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.com

2. resolved Aug 25, 2026, 03:04 PM UTC

   This incident has been resolved. Between 01:42 and 02:02 UTC-5, monitoring detected a single failing request out of 118 total checks (117 passing, 0 degraded). Average response time remained stable at 546 ms. The failure was isolated to one query and did not represent a systemic issue or a sustained service disruption. No further failing or degraded requests have been observed since, and the API is operating normally.

Read the full incident report →

Started Aug 19, 2026, 08:37 PM UTC· resolved Aug 19, 2026, 09:35 PM UTC

1. identified Aug 19, 2026, 08:37 PM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.com

2. resolved Aug 19, 2026, 09:35 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 19, 2026, 08:37 PM UTC· resolved Aug 20, 2026, 02:46 AM UTC

1. identified Aug 19, 2026, 08:37 PM UTC

   An issue was found in AGENT. Click for details: https://availability.fluidattacks.com

2. resolved Aug 20, 2026, 02:46 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 18, 2026, 02:00 PM UTC· resolved Aug 18, 2026, 02:00 PM UTC

1. resolved Aug 18, 2026, 02:00 PM UTC

   Some of our automated flows are currently failing: repository cloning, reattacks on automatic reports, and SBOM requests are not completing. The cause has been traced to an issue in our infrastructure, and our team is already working on restoring these flows. Existing information remains complete and intact, and there is no impact to security or to any other part of the platform.

2. postmortem Aug 20, 2026, 04:27 AM UTC

   ### Postmortem #### Impact Some users experienced failures in several of our automated flows: repositories could not be cloned, reattacks on automatic reports did not run, and SBOM requests were not fulfilled. The issue started on UTC-5 26-08-18 09:07 and was reactively discovered 6 hours \(TTD\) later by a staff member from the hacking team who noticed that reattacks on automatic reports had been queued for more than three hours without progressing. The problem was resolved in 1.2 hours \(TTF\), resulting in a total window of exposure of 7.2 hours \(WOE\). * **Repository cloning**: cloning executions did not run, so repository contents were not refreshed during the window. * **Reattacks on automatic reports**: reattack requests were accepted but stayed queued without being executed. * **SBOM requests**: requested SBOM files were not generated or delivered. #### Cause These three flows are not run directly by the platform. The platform builds and enqueues an execution, and a set of state machines then carries out the actual work; the handoff happens at a stage called `batch_dispatch`. A change updated the definition of those state machines, along with the components downstream of them, to a new format for passing the parameters of each execution[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/110512).) The component that sits upstream, the one inside the platform that generates and enqueues the executions, was not updated in the same change, so it kept dispatching against the previous version. From that point on, the parameters it sent no longer matched what the state machines expected, and every execution it enqueued failed at the handoff stage instead of running. The mismatch was not caught before reaching production because this kind of infrastructure change is not currently covered by an end-to-end test in our continuous integration pipeline, and the developer making the change did not have full visibility into how the two sides of the flow depend on each other. #### Solution The version of the state machines referenced by the platform's dispatching component was updated to the latest one, bringing both sides of the flow back into agreement[.](https://gitlab.com/fluidattacks/universe/-/merge_requests/111060).) Cloning, reattack and SBOM executions resumed processing immediately afterwards, including the work that had accumulated while the flows were failing. To prevent a recurrence, the team will evaluate testing alternatives that can detect this class of version mismatch from the continuous integration pipeline, before a change of this kind reaches production. #### Conclusion This incident showed that a change to the state machine definitions and a change to the component that dispatches their executions must be rolled out together, and that nothing in our pipeline currently verifies that the two remain compatible. An end-to-end test covering the dispatch path, together with clearer documentation of how the flow works from end to end, would have surfaced the mismatch before any user was affected. **MISSING\_TEST < COMMUNICATION\_FAILURE < INCOMPLETE\_PERSPECTIVE**

Read the full incident report →

Started Aug 16, 2026, 11:11 PM UTC· resolved Aug 18, 2026, 11:08 PM UTC

1. identified Aug 16, 2026, 11:11 PM UTC

   An issue was found in API. Click for details: https://availability.fluidattacks.com

2. resolved Aug 18, 2026, 11:08 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 06, 2026, 11:00 PM UTC· resolved Aug 06, 2026, 11:00 PM UTC

1. resolved Aug 06, 2026, 11:00 PM UTC

   Some groups are experiencing cloning failures on repositories configured with ZTNA or Egress connectivity. Affected repositories stay in a "Cloning queued..." state without completing, which prevents our automated scanners from retrieving recent changes and leaves vulnerability reporting for those repositories outdated. The cause has been identified in our connector infrastructure and a fix is underway.

2. postmortem Aug 13, 2026, 05:12 AM UTC

   ## Postmortem ### Impact At least one user experienced automated cloning failures on repositories configured with ZTNA or Egress connectivity, which left the affected roots stuck in a `QUEUED` state and prevented automated scanners from retrieving recent changes, causing vulnerability reporting for those repositories to go stale. The issue started on 2026-08-06 at 17:48 \(UTC-5\) and was reactively discovered 1.9 days later by a staff member who received a help-desk report that a customer's root had been stuck in a "Cloning queued..." state for more than two hours with no vulnerabilities being displayed. The problem was resolved within an additional 1.8 days, resulting in a total window of exposure of 3.7 days. 26 groups were identified with affected repositories. ### Cause As part of a scheduled secrets rotation cycle, the WARP connector credentials were rotated to values that were not valid for the connector's registration. The rotation procedure documented for this credential pair was not followed, and the new values were never validated against the connector before being deployed. With an invalid registration, the connector could not establish the tunnel to the customer network prior to cloning, and every cloning job on the affected queue failed. Two properties of the platform kept the failure invisible: the connector error was unhandled, so the root's cloning status was never transitioned out of `QUEUED` and the interface continued to display "Cloning queued..." instead of a failure state; and a credential error inside the root-filtering step caused cloning to abort for every root in the group rather than only those bound to the broken credential. No alert was raised for the sustained cloning failures, so the outage only became visible when a customer escalation reached the help desk. ### Solution The credential rotation was reverted to the previous valid value, restoring the connector's registration and unblocking cloning for the affected roots. Cloning was then hardened so that failures of this class cannot be silent again: group cloning actions were wrapped in a general exception handler so that an unexpected failure in one group no longer blocks the rest; roots are now partitioned by credential health before validation, so a credential error marks only its own roots as failed instead of aborting the entire group gather; and logging was upgraded to emit real tracebacks on failures. A proper rotation process for this credential pair will be established with tighter controls and a step-by-step guide, and sustained cloning failures will now trigger an explicit alert. ### Conclusion A credential rotation performed outside its documented procedure disabled the connector's network registration. Because the resulting cloning failures were neither surfaced in the root status nor alerted on, the outage persisted until a customer escalated it. Validating rotated credentials against the services that consume them before deployment, and alerting on sustained cloning failures, would have contained this within hours instead of days. **ROTATION\_FAILURE < MISSING\_ALERT < INCOMPLETE\_PERSPECTIVE**

Read the full incident report →

Started Jul 21, 2026, 05:13 PM UTC· resolved Jul 22, 2026, 07:30 PM UTC

1. investigating Jul 21, 2026, 05:13 PM UTC

   The analytics charts (by group, by organization, and portfolio views) are currently displaying outdated information. The data behind these charts last refreshed on 2026-06-24 and has not updated since, so recent activity, including groups and repositories created after that date, is not yet reflected. This issue is limited to the charts only: the underlying data remains complete, intact, and unaffected, and there is no impact to security or to any other part of the platform. Our team is actively investigating the cause and working to restore regular chart updates. Further updates will be posted as we make progress.

2. resolved Jul 22, 2026, 07:30 PM UTC

   This incident has been resolved.

3. postmortem Jul 24, 2026, 01:51 AM UTC

   ### Postmortem ### Impact Users experienced stale analytics charts across the platform: the documents powering the "by group", "by organization", and "portfolio" views stopped being refreshed, leaving the vast majority of organizations displaying outdated data as of June 24. Groups and repositories created after that date did not appear in the charts even when data was available. The issue started on June 23, 2026 and was discovered approximately 20 days later by a staff member who noticed the charts were not up to date. Once identified, the problem was fully resolved within 1 day. ### Cause A scheduled background job responsible for regenerating analytics charts was failing silently. When processing a large number of organizations, a single error during the generation of one organization's charts caused the entire job to stop, preventing all remaining organizations from being updated. Over time, increased load on the platform's data layer began triggering these errors more frequently and across more organizations, amplifying the impact. Because no alerts were configured to notify the team when this job failed, the issue went undetected for several weeks. ### Solution The chart generation process was updated to handle errors on a per-organization basis: if one organization's charts fail to generate, it is logged and skipped, and processing continues for all others. Additional safeguards were introduced to reduce pressure on the underlying data layer during high-load periods. Alerting was also configured so that failures in this job are now immediately surfaced to the engineering team, preventing similar silent failures in the future. ### Conclusion A background job with no error isolation and no failure alerting can silently affect customer-facing data for an extended period before anyone notices. The key improvements, isolating errors per subject and adding alerts on job failures, ensure this class of issue will be detected and addressed much faster going forward. We apologize for the disruption and are committed to maintaining the accuracy and freshness of your analytics data.

Read the full incident report →

Started Jul 09, 2026, 03:55 PM UTC· resolved Jul 09, 2026, 05:50 PM UTC

1. investigating Jul 09, 2026, 03:55 PM UTC

   Some groups are receiving unexpected open events related to environment URL availability. These events were generated incorrectly and do not reflect actual issues in the affected groups. Our team is investigating and working on a fix.

2. resolved Jul 09, 2026, 05:50 PM UTC

   This incident has been resolved.

3. postmortem Jul 10, 2026, 04:51 AM UTC

   ### Postmortem #### Impact At least one user experienced a flood of false-positive `ENVIRONMENT_ISSUES` events on their production groups, each dispatching an automatic email notification, for URL environments that were in fact reachable. The issue started on 2026-07-08 at 20:00 \(UTC-5\) and was reactively discovered 14.9 hours later by a staff member who noticed an unusually high volume of events of this type opened simultaneously across multiple production groups, all sharing an identical "URL is not reachable" description. The problem was resolved within an additional 2.6 hours, resulting in a total window of exposure of 17.5 hours. #### Cause A new URL environment connectivity check ran for the first time in production on its automated schedule. The logic that determines which URLs are eligible for the check contained a flaw: instead of correctly excluding URLs that require authentication or that are only reachable through special network channels, it included them in an unauthenticated probe. Upon receiving unauthorized-access responses, such as redirects to login portals, the system incorrectly classified those URLs as unreachable and opened an incident event for each one, also dispatching a notification email to the group's stakeholders. The affected URLs were fully functional; they simply required credentials that the automated probe never supplied. Approximately 850 spurious events were created across production groups. #### Solution The approximately 850 incorrectly generated events were closed through an automated process that identified them by type, description, and creation time, and closed them directly without emitting further notification emails to users. The automated scheduler was disabled to prevent the issue from recurring. As a permanent fix, the connectivity check will no longer run automatically and will instead become an on-demand action that internal users can trigger manually from the interface. Additionally, the eligibility logic will be corrected to properly exclude authenticated environments and those operating through special network channels, and an exhaustive dry-run against real production data will be required before any re-enablement. #### Conclusion This incident revealed that the URL eligibility logic and the handling of authenticated environments were never validated end-to-end before the first production run, allowing an unauthenticated probe to be misread as a connectivity failure at scale. Tests covering the eligibility logic and the authenticated-URL flow, along with a dry-run against real production data before enabling any automated task, would have prevented these events from being created. **MISSING\_TEST < INCOMPLETE\_PERSPECTIVE**

Read the full incident report →

Started Jun 18, 2026, 09:30 PM UTC· resolved Jun 18, 2026, 11:42 PM UTC

1. identified Jun 18, 2026, 09:30 PM UTC

   An issue was found in FIXES-CODE. Click for details: https://availability.fluidattacks.com

2. resolved Jun 18, 2026, 11:42 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 26, 2026, 08:28 PM UTC· resolved May 26, 2026, 09:26 PM UTC

1. investigating May 26, 2026, 08:28 PM UTC

   Users may experience an unexpected error when accessing the Evidence page. The application fails to load due to a null reference issue while processing evidence data. Our team is investigating and working on a fix.

2. identified May 26, 2026, 08:59 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 26, 2026, 09:17 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 26, 2026, 09:26 PM UTC

   The issue causing the Evidence page to crash has been identified and resolved. The page is now loading correctly for all users. We apologize for the disruption and appreciate your patience while we worked through it.

5. postmortem May 30, 2026, 05:45 AM UTC

   ## Impact All users on the platform experienced the inability to view or upload evidence on finding pages for approximately 58 minutes. Both the web interface and the desktop application were affected. Users without access to certain findings were additionally stuck in a reload loop during the same window. The issue started on UTC-5 26-05-26 14:35 and was proactively discovered 57.6 minutes \(TTD\) later by a staff member who noticed that evidence data was not loading and alerted the team. The problem was resolved in 57.6 minutes \(TTF\), resulting in a total window of exposure of 1.9 hours \(WOE\). ## Cause A planned dependency upgrade deployed to remediate reported security vulnerabilities introduced an undocumented behavioral change in a core library responsible for mapping GraphQL field names to their internal data keys. Under the previous version, fields whose names were entirely lowercase were left unchanged during this mapping. Under the upgraded version, all fields, regardless of casing were passed through a name converter that inserts underscores before numeric characters. As a result, fields such as those used to store evidence items were silently redirected to keys that did not exist in the underlying data, causing them to return empty values instead of their content. The behavioral change was documented in the library's internal pull request history but was not prominently surfaced in the official changelog, and the existing test suite did not cover the full name-resolution flow against a live schema. ## Solution An emergency rollback to the previous dependency versions was deployed the same day, restoring all affected functionality within the incident window. A definitive fix was subsequently developed and deployed two days later: the dependency upgrade was re-applied alongside a custom name-mapping configuration that preserves the correct resolution behavior for all-lowercase field names while still applying the standard conversion to mixed-case names. Automated tests were added to explicitly validate that all numbered evidence fields resolve to their expected values, preventing silent regressions of this class in future dependency upgrades. ## Conclusion Major-version upgrades to core GraphQL infrastructure libraries must be accompanied by integration tests that exercise the full field-name resolution pipeline against a real executable schema not only unit tests of business logic. The failure was silent by design: fields returned empty values without raising errors, making it indistinguishable from missing data until users reported it. Future upgrades to schema-handling libraries require an explicit end-to-end assertion that no field resolves to null when data is present. **THIRD\_PARTY\_CHANGE < MISSING\_TEST < INCOMPLETE\_PERSPECTIVE**

Read the full incident report →

Started May 20, 2026, 01:37 PM UTC· resolved May 20, 2026, 02:54 PM UTC

1. identified May 20, 2026, 01:37 PM UTC

   An issue was found in DOCS. Click for details: https://availability.fluidattacks.com

2. identified May 20, 2026, 02:32 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved May 20, 2026, 02:54 PM UTC

   This incident has been resolved.

4. postmortem May 21, 2026, 04:40 AM UTC

   ## Impact On May 20, 2026, users visiting [docs.fluidattacks.com](http://docs.fluidattacks.com/) may have encountered a "Server failed to respond" error for approximately 1.3 hours. The issue was detected internally by our team and resolved within the same window. ## What happened During a routine deployment operation, an outdated version of the documentation service was accidentally restored to production. This older version was missing a configuration component required for the service to start up correctly, which caused the site to become unavailable. Think of it like accidentally restoring an old backup that was missing a key setting the service came back up in an incomplete state and was unable to respond to requests. ## What we changed Service was restored by redeploying the latest stable version of the documentation site, bringing `docs.fluidattacks.com` back online. We then applied a permanent fix to ensure the required configuration is always present and correctly loaded when the service starts, preventing the same failure mode from recurring. ## Going forward We have updated our deployment process to prevent outdated versions from being restored to production. Safeguards are now in place to ensure that only verified, fully configured versions of the service can be deployed. **COMMUNICATION\_FAILURE < INCOMPLETE\_PERSPECTIVE.**

Read the full incident report →

Started May 19, 2026, 11:57 PM UTC· resolved May 20, 2026, 02:31 AM UTC

1. identified May 19, 2026, 11:57 PM UTC

   An issue was found in DOCS. Click for details: https://availability.fluidattacks.com

2. resolved May 20, 2026, 02:31 AM UTC

   This incident has been resolved.

3. postmortem May 21, 2026, 04:18 AM UTC

   # Postmortem ## Impact On May 19, 2026, users visiting docs.fluidattacks.com may have encountered a "Server failed to respond" error for approximately 2.6 hours. The issue was detected internally by our team and resolved within the same window. ## What happened During a routine deployment operation, an outdated version of the documentation service was accidentally restored to production. This older version was missing a configuration component required for the service to start up correctly, which caused the site to become unavailable. Think of it like accidentally restoring an old backup that was missing a key setting the service came back up in an incomplete state and was unable to respond to requests. ## What we changed Service was restored by redeploying the latest stable version of the documentation site, bringing `docs.fluidattacks.com` back online. We then applied a permanent fix to ensure the required configuration is always present and correctly loaded when the service starts, preventing the same failure mode from recurring. ## Going forward We have updated our deployment process to prevent outdated versions from being restored to production. Safeguards are now in place to ensure that only verified, fully configured versions of the service can be deployed. **COMMUNICATION\_FAILURE < INCOMPLETE\_PERSPECTIVE.**

Read the full incident report →