Is Crazy Ant Labs down?

A fix has been implemented and we are monitoring the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## What happened A platform update released on the afternoon of 16 August introduced two faults. One stopped our services reusing internal connections, so under load a share of requests failed. The other stopped the background process that finishes writing activity records. The service stayed available and most requests succeeded, but a meaningful number did not — and our systems reported those failures in a way that made them harder to detect than they should have been. ## What you may have experienced * Being told your **sign-in details were incorrect**, or that **single sign-on was not enabled**, when neither was true * Intermittent errors, or actions that failed without a clear reason * **Webhook notifications that were never sent** for some file activity * Automations that did not run * **Activity log entries missing or incomplete**, and last-login times not updating Retrying generally worked. The webhook and activity-log items are the exceptions. ## Your data **No files were lost, altered, or exposed**, and **no unauthorized access occurred** — where the fault touched access decisions it refused legitimate requests; it never granted access to anyone. Transfers themselves were unaffected: what could go missing was the notification about a file, not the file. **Webhooks.** For some file activity no notification was created, so your endpoint was never called. These are not retried automatically. A minority of organizations that use webhooks were affected, and for most of them a minority of events. We can identify which files this affected for your organization specifically, from storage records that are independent of the fault. If you rely on webhooks, contact support and we will send you the list — full paths and file sizes — so the events can be reconciled or replayed on your side. **Activity logs.** Two things happened. Most entries were written but left incomplete — the location and device detail added after the fact is missing. That has now been fully restored, because it derives from information the entries still hold. Last-login times and login counts, however, cannot be reliably restated. Separately, **a substantial number of entries were not recorded at all** — a significant share of the activity in that window, concentrated in connection and authentication records. We can tell you the exact figure for your organization. **We are restoring these too.** The underlying activity was captured in our infrastructure logs independently of the fault, so the great majority of the missing entries can be rebuilt from their original source rather than reconstructed or estimated. That work is underway and we expect it to complete within the next few days. Restored entries will be marked internally as recovered after the fact, and will carry both their original event time and the time they were rebuilt, so the record stays transparent about what happened. A small remainder — administrative actions taken through the web app — has no independent source and cannot be recovered. That is a genuine gap. If you are subject to an audit covering this period, contact us before relying on the log for it, and we will tell you exactly what is covered, what is being restored, and what is not. ## What we have done Both faults are fixed and live in production. The missing detail on existing entries has been fully restored. Rebuilding the entries that were never recorded is underway and expected to complete within the next few days. ## What we are changing * **Temporary failures will be reported as failures.** Some internal errors were reported as though the thing being looked up did not exist — which is why a sign-in problem surfaced as "incorrect credentials", and why our alerting did not fire early enough. * **Alerting on this class of fault**, so a background process that stops producing output is caught in minutes rather than hours. * **Pre-release verification extended** to cover faults that only appear against real infrastructure under load, which is why our automated tests missed both. ## Questions If you require specific details regarding your organization's impact, please contact support and reference this incident.