Crazy Ant Labs Outage History

Started Aug 17, 2026, 10:34 AM UTC· resolved Aug 17, 2026, 02:57 PM UTC

1. investigating Aug 17, 2026, 10:34 AM UTC

   A fix has been implemented and we are monitoring the issue.

2. monitoring Aug 17, 2026, 12:14 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 17, 2026, 02:57 PM UTC

   This incident has been resolved.

4. postmortem Aug 18, 2026, 08:13 AM UTC

   ## What happened A platform update released on the afternoon of 16 August introduced two faults. One stopped our services reusing internal connections, so under load a share of requests failed. The other stopped the background process that finishes writing activity records. The service stayed available and most requests succeeded, but a meaningful number did not — and our systems reported those failures in a way that made them harder to detect than they should have been. ## What you may have experienced * Being told your **sign-in details were incorrect**, or that **single sign-on was not enabled**, when neither was true * Intermittent errors, or actions that failed without a clear reason * **Webhook notifications that were never sent** for some file activity * Automations that did not run * **Activity log entries missing or incomplete**, and last-login times not updating Retrying generally worked. The webhook and activity-log items are the exceptions. ## Your data **No files were lost, altered, or exposed**, and **no unauthorized access occurred** — where the fault touched access decisions it refused legitimate requests; it never granted access to anyone. Transfers themselves were unaffected: what could go missing was the notification about a file, not the file. **Webhooks.** For some file activity no notification was created, so your endpoint was never called. These are not retried automatically. A minority of organizations that use webhooks were affected, and for most of them a minority of events. We can identify which files this affected for your organization specifically, from storage records that are independent of the fault. If you rely on webhooks, contact support and we will send you the list — full paths and file sizes — so the events can be reconciled or replayed on your side. **Activity logs.** Two things happened. Most entries were written but left incomplete — the location and device detail added after the fact is missing. That has now been fully restored, because it derives from information the entries still hold. Last-login times and login counts, however, cannot be reliably restated. Separately, **a substantial number of entries were not recorded at all** — a significant share of the activity in that window, concentrated in connection and authentication records. We can tell you the exact figure for your organization. **We are restoring these too.** The underlying activity was captured in our infrastructure logs independently of the fault, so the great majority of the missing entries can be rebuilt from their original source rather than reconstructed or estimated. That work is underway and we expect it to complete within the next few days. Restored entries will be marked internally as recovered after the fact, and will carry both their original event time and the time they were rebuilt, so the record stays transparent about what happened. A small remainder — administrative actions taken through the web app — has no independent source and cannot be recovered. That is a genuine gap. If you are subject to an audit covering this period, contact us before relying on the log for it, and we will tell you exactly what is covered, what is being restored, and what is not. ## What we have done Both faults are fixed and live in production. The missing detail on existing entries has been fully restored. Rebuilding the entries that were never recorded is underway and expected to complete within the next few days. ## What we are changing * **Temporary failures will be reported as failures.** Some internal errors were reported as though the thing being looked up did not exist — which is why a sign-in problem surfaced as "incorrect credentials", and why our alerting did not fire early enough. * **Alerting on this class of fault**, so a background process that stops producing output is caught in minutes rather than hours. * **Pre-release verification extended** to cover faults that only appear against real infrastructure under load, which is why our automated tests missed both. ## Questions If you require specific details regarding your organization's impact, please contact support and reference this incident.

Read the full incident report →

Started Dec 29, 2025, 11:52 AM UTC· resolved Dec 29, 2025, 01:47 PM UTC

1. investigating Dec 29, 2025, 11:52 AM UTC

   We are currently investigating this issue.

2. investigating Dec 29, 2025, 11:55 AM UTC

   We are continuing to investigate this issue.

3. identified Dec 29, 2025, 01:00 PM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Dec 29, 2025, 01:47 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 15, 2025, 02:45 PM UTC· resolved Dec 16, 2025, 05:32 AM UTC

1. investigating Dec 15, 2025, 02:45 PM UTC

   We are currently investigating this issue.

2. investigating Dec 15, 2025, 03:04 PM UTC

   We are continuing to investigate this issue.

3. investigating Dec 15, 2025, 03:48 PM UTC

   We are continuing to investigate this issue.

4. identified Dec 15, 2025, 05:05 PM UTC

   The issue has been identified and a fix is being implemented.

5. resolved Dec 16, 2025, 05:32 AM UTC

   Earlier today, one of our US-based SFTP / FTPS servers (sparkling-water-50295.sftptogo.com) experienced a very high volume of active connections, which led to throttling errors for a limited number of clients attempting to connect. We have resolved the issue, and full connectivity and functionality have been restored. The root cause was determined to be an overload of concurrent connections on the affected server. We continue to monitor the server closely to ensure stability and prevent recurrence. No further impact is observed at this time. Thank you for your patience. If you're still experiencing any issues, please don't hesitate to contact our support team.

Read the full incident report →

Started Oct 20, 2025, 10:35 AM UTC· resolved Oct 20, 2025, 10:40 PM UTC

1. monitoring Oct 20, 2025, 10:35 AM UTC

   Due to AWS's incident the following components are still impacted, SFTP To Go's audit logs and Cron To Go's processing engine

2. monitoring Oct 20, 2025, 10:53 AM UTC

   We are continuing to monitor for any further issues.

3. monitoring Oct 20, 2025, 05:02 PM UTC

   Due to AWS's incident the following components are still impacted, SFTP To Go's audit logs and SSO authentication and Cron To Go's processing engine. Support chat is lagging so new tickets are slow to get a response.

4. monitoring Oct 20, 2025, 06:11 PM UTC

   Please use this server host name to connect summer-wind-35319.sftptogo.com. This server name is very secure and many FTP clients that lack the newest security algorithms will not be able to connect to it. Due to AWS's incident the following components are still impacted, SFTP To Go's audit logs and SSO authentication and Cron To Go's processing engine. Support chat is lagging so new tickets are slow to get a response.

5. resolved Oct 20, 2025, 10:40 PM UTC

   Our services were impacted by the large event of our cloud provider. The incident would have caused varying types of degradation of our services - inability to log in to our dashboard, increased error rates in API calls, high latency, and other services, integrations and channels not working. During this incident, our Engineering teams worked to work through all processing back logs and rebalance the service through other available regions. This incident is now resolved, and all services are now working as expected.

Read the full incident report →

Started Jun 10, 2025, 10:15 AM UTC· resolved Jun 11, 2025, 06:13 AM UTC

1. investigating Jun 10, 2025, 10:15 AM UTC

   We are currently investigating this issue.

2. investigating Jun 10, 2025, 10:17 AM UTC

   We are waiting for news from Heroku regarding these issues.

3. monitoring Jun 10, 2025, 04:47 PM UTC

   We are seeing continued recovery across Heroku APIs. We will provide further updates when available.

4. resolved Jun 11, 2025, 06:13 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 09, 2025, 12:24 PM UTC· resolved Jun 09, 2025, 01:12 PM UTC

1. investigating Jun 09, 2025, 12:24 PM UTC

   We are currently investigating this issue.

2. investigating Jun 09, 2025, 12:36 PM UTC

   We are continuing to investigate this issue.

3. resolved Jun 09, 2025, 01:12 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 29, 2025, 06:30 PM UTC· resolved Apr 29, 2025, 08:41 PM UTC

1. investigating Apr 29, 2025, 06:30 PM UTC

   We identified an issue with Activity To Go subscription events. Users may experience a delay receiving Heroku events. Our team is investigating the root cause.

2. identified Apr 29, 2025, 06:32 PM UTC

   Our team has identified an issue with authenticating Heroku Webhook requests which triggers events. The team is working on a fix.

3. identified Apr 29, 2025, 07:58 PM UTC

   The team applied a fix and event deliveries backlog are being delivered.

4. monitoring Apr 29, 2025, 08:34 PM UTC

   The events backlog is still being caught up.

5. resolved Apr 29, 2025, 08:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2025, 12:15 PM UTC· resolved Feb 19, 2025, 04:20 PM UTC

1. investigating Feb 19, 2025, 12:15 PM UTC

   We are currently investigating this issue.

2. resolved Feb 19, 2025, 04:20 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 21, 2024, 04:38 PM UTC· resolved Nov 21, 2024, 06:14 PM UTC

1. investigating Nov 21, 2024, 04:38 PM UTC

   We are currently investigating this issue.

2. identified Nov 21, 2024, 05:34 PM UTC

   The issue has been identified and is caused by a Chrome browser version update that changes the path to the local file in upload. Other browsers (e.g. Firefox, Safari) seem to be unaffected. The team is looking for a workaround.

3. resolved Nov 21, 2024, 06:14 PM UTC

   This incident has been resolved.

Read the full incident report →