Scanii.com Outage History

Started Oct 21, 2026, 10:49 PM UTC· resolved Oct 21, 2026, 11:29 PM UTC

1. identified Oct 21, 2026, 10:49 PM UTC

   Our monitoring system has identified issues with the region and our engineers have been dispatch to investigate.

2. resolved Oct 21, 2026, 11:29 PM UTC

   Our automated monitors have identified a problem with this component and our engineers have been dispatched to investigate.

Read the full incident report →

Started Oct 12, 2026, 06:00 PM UTC· resolved Oct 12, 2026, 06:03 PM UTC

1. identified Oct 12, 2026, 06:00 PM UTC

   Our monitoring system has identified issues with the region and our engineers have been dispatch to investigate.

2. resolved Oct 12, 2026, 06:03 PM UTC

   Our automated monitors have identified a problem with this component and our engineers have been dispatched to investigate.

Read the full incident report →

Started Aug 11, 2026, 08:38 AM UTC· resolved Aug 11, 2026, 09:28 AM UTC

1. investigating Aug 11, 2026, 08:38 AM UTC

   We are seeing an elevated error rate for this region, engineering is looking into it.

2. resolved Aug 11, 2026, 09:28 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 13, 2026, 01:59 PM UTC· resolved Feb 13, 2026, 09:59 PM UTC

1. monitoring Feb 13, 2026, 01:59 PM UTC

   Results are taking longer than usual to be available in our administrator portal. This does not impact actual file processing.

2. resolved Feb 13, 2026, 09:59 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 04, 2026, 02:58 PM UTC· resolved Feb 04, 2026, 03:11 PM UTC

1. identified Feb 04, 2026, 02:58 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Feb 04, 2026, 03:11 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 24, 2026, 01:50 AM UTC· resolved Jan 24, 2026, 02:37 AM UTC

1. identified Jan 24, 2026, 01:50 AM UTC

   Our monitors have identified an issue with our administrative portal (www.scanii.com) and our engineers were dispatched to address it. This does not impact content processing.

2. identified Jan 24, 2026, 02:29 AM UTC

   Content ID results are delayed, we are working on a fix.

3. resolved Jan 24, 2026, 02:37 AM UTC

   The incident has been resolved and service fully restored.

Read the full incident report →

Started Dec 23, 2025, 08:21 AM UTC· resolved Dec 23, 2025, 08:24 AM UTC

1. investigating Dec 23, 2025, 08:21 AM UTC

   Our monitoring system has identified issues with the region and our engineers have been dispatch to investigate.

2. resolved Dec 23, 2025, 08:24 AM UTC

   The incident has been resolved and service fully restored.

Read the full incident report →

Started Dec 12, 2025, 01:07 PM UTC· resolved Dec 12, 2025, 01:23 PM UTC

1. identified Dec 12, 2025, 01:07 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Dec 12, 2025, 01:20 PM UTC

   We identified a couple failing instances and have removed them from the pool

3. resolved Dec 12, 2025, 01:23 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 09:05 AM UTC· resolved Oct 20, 2025, 10:55 PM UTC

1. investigating Oct 20, 2025, 01:05 PM UTC

   Processing is behind across multiple regions, we are investigating and will provide an update as soon as possible. At this point we believe this is fall out from the major AWS outage earlier

2. investigating Oct 20, 2025, 01:06 PM UTC

   We are continuing to investigate this issue.

3. investigating Oct 20, 2025, 01:07 PM UTC

   We are continuing to investigate this issue.

4. identified Oct 20, 2025, 01:29 PM UTC

   The issue is primarily caused by SQS failing in us-east-1, that is unfortunately also impacting other regions that submit consumption data to us-east-1

5. identified Oct 20, 2025, 01:44 PM UTC

   We are also unable to harvest compute capacity in us-east-1, so we're at AWS's mercy in order to fully recover. It looks like they are actively working to restore EC2

6. identified Oct 20, 2025, 03:56 PM UTC

   We are continuing to work on a fix for this issue.

7. monitoring Oct 20, 2025, 03:57 PM UTC

   Still ongoing

8. monitoring Oct 20, 2025, 07:33 PM UTC

   We are now able to scale again and are processing through a large backlog of fetch/asynchronous work. Synchronous analysis calls should be mostly restored to normal - for US1 All other regions should be working normally now

9. monitoring Oct 20, 2025, 07:53 PM UTC

   We are continuing to monitor for any further issues.

10. resolved Oct 20, 2025, 10:55 PM UTC

    This incident has been resolved.

11. postmortem Oct 27, 2025, 01:45 PM UTC

    # Postmortem 2025-10-20 Outage ### Amazon Service Disruption in the Northern Virginia \(US-EAST-1\) Region ### What happened **Impact** * **Duration:** 10.5 hours \(5:30 AM – 4:00 PM EDT\) * **Affected regions:** us1, eu1, eu2, ap1, ap2, ca1 Between 5:30 AM and 4:00 PM EDT on October 20 2025, Scanii services were globally unavailable or severely degraded. The following timeline summarizes key events \(all times EDT\): 1. **3:11 AM:** AWS Health reported increased error rates and latencies in us-east-1. 2. **4:26 AM:** AWS expanded the incident to include connectivity issues with DynamoDB. Because DynamoDB is foundational, multiple AWS services were also affected. 3. **5:01 AM:** AWS stated that the root cause had been identified and mitigated. 4. **5:30 AM:** Our synthetic monitoring detected that [www.scanii.com](https://www.scanii.com/) was down. The on-call engineer found ECS service capacity at zero, noted AWS recovery in progress, started new Fargate tasks, observed recovery, and stood down. 5. **6:35 AM:** AWS reported that EC2 launches were still experiencing elevated error rates, worsening the impact. 6. **9:00 AM:** Multiple customer support tickets indicated that the service was unavailable across regions. Despite no active alerts, we opened an incident and backdated it to the first alert. 7. **9:30 AM:** We traced the root cause to AWS SQS in us-east-1. The failure prevented job completion messages \(used for consumption tracking\) from other regions from reaching us-east-1, causing global instability. Our monitoring—recently migrated to AWS—was also affected, explaining the missing alerts. 8. **11:04 AM:** AWS reported ongoing network connectivity issues across multiple services. 9. **11:30 AM:** We considered declaring a disaster and invoking our DR plan to migrate workloads from us-east-1 to another U.S. region. Although the plan is tested twice a year, this would have been its first live execution and an irreversible migration. Given AWS’s ongoing progress and the historically short duration of their incidents, we decided against activating it. 10. **3:33 PM:** All regions except us1 had recovered, but us1 still lacked EC2 capacity to process the backlog. 11. **4:00 PM:** EC2 capacity recovered; we scaled up and achieved full service restoration. AWS later issued a [post-incident summary](https://aws.amazon.com/message/101925/) attributing the event to DNS resolution issues affecting DynamoDB. Scanii’s direct root cause was SQS unavailability in us-east-1, a shared dependency across all regions. ### Why it took 3.5 hours \(5:30 AM – 9:00 AM\) to detect multi-region impact 1. Failures in regions outside us-east-1 were intermittent and below alert thresholds. 2. Alerting in us-east-1 was impaired because synthetic monitors were hosted inside the affected region. ### Preventive actions Because this event represents the largest downtime in company history \(including the 2017 S3 outage\), we are implementing the following corrective measures: 1. **Backup monitoring outside AWS:** We are building an external monitoring infrastructure independent of AWS. Our previous Azure-based system was retired earlier this year when the product was discontinued, leaving us temporarily dependent on AWS for monitoring. 2. **Alerting audit:** We are reviewing and tightening CloudWatch monitors to better capture transient and partial failures. 3. **Cross-region architecture:** We are re-architecting our billing and consumption system so a failure in us-east-1 cannot affect other regions. This project will begin in Q1 2026. 4. **Regional relocation evaluation:** We are assessing a migration of selected services from us-east-1 to us-east-2 \(Ohio\). Historical evidence suggests us-east-1 experiences more frequent regional-scale incidents. Any migration will require planned downtime and validation of available capacity in the destination region. ### Customer communication We apologize for the disruption. Customers affected by this incident may request prorated credits under our SLA: [https://docs.scanii.com/article/141-sla](https://docs.scanii.com/article/141-sla). For questions, contact [[email protected]](mailto:[email protected]).

Read the full incident report →

Started Oct 16, 2025, 08:47 PM UTC· resolved Oct 16, 2025, 08:53 PM UTC

1. investigating Oct 16, 2025, 08:47 PM UTC

   Our monitoring system has identified issues with the region and our engineers have been dispatch to investigate.

2. resolved Oct 16, 2025, 08:53 PM UTC

   The incident has been resolved and service fully restored.

Read the full incident report →

Started May 15, 2025, 07:18 PM UTC· resolved May 15, 2025, 07:34 PM UTC

1. investigating May 15, 2025, 07:18 PM UTC

   Our monitoring system has identified issues with the region and our engineers have been dispatch to investigate.

2. resolved May 15, 2025, 07:34 PM UTC

   The incident has been resolved and service fully restored.

Read the full incident report →

Started May 14, 2025, 04:00 PM UTC· resolved May 15, 2025, 11:30 AM UTC

1. investigating May 15, 2025, 11:15 AM UTC

   We are investigating issues with users accessing our web portal at www.scanii.com

2. investigating May 15, 2025, 11:29 AM UTC

   Looks like we're back up, I'll provide a post mortem as soon as we can.

3. resolved May 15, 2025, 11:30 AM UTC

   This incident has been resolved.

4. postmortem May 16, 2025, 01:00 PM UTC

   ### Summary Between 2025-05-14 14:00 UTC and 2025-05-15 10:00 UTC, Scanii’s administrative portal \([www.scanii.com](https://www.scanii.com/)\) experienced an outage lasting approximately 18 hours. The downtime was caused by a misconfiguration in our Terraform infrastructure code, which unintentionally triggered the replacement of critical services in the AWS us-east-1 region. This disruption should have been promptly caught by our monitoring systems. However, two separate issues prevented timely detection and alerting, compounding the impact of the incident. ### **Root Cause** **1** **Terraform Misconfiguration**: A change in our infrastructure-as-code repository inadvertently marked several key services for replacement. This change was not caught during our usual review processes. **2** **Monitoring Failures**: * **Azure Monitor**: A prior configuration update — made in response to the retirement of an Azure availability feature — changed the threshold logic for incident creation. This update set an unattainable condition, effectively disabling alerts from Azure's monitoring system. * **CloudWatch Integration**: An unrelated change to improve internal alerting \(e.g., backup system monitoring\) broke the CloudWatch → PagerDuty integration, preventing critical alarms from notifying our on-call team. ### ⠀Impact * Portal access was unavailable for ~18 hours. * No customer API traffic was affected. ### **Remediation and Next Steps** To prevent recurrence, we are taking the following actions: **1** **Pre-Production Alerting Tests**: We will implement automated, routine alert testing in our pre-production environment to validate that alert paths are functional. **2** **Monitoring Infrastructure Audit**: We are conducting a full audit of all alerting and monitoring infrastructure-as-code configurations to identify and resolve similar misconfigurations. **3** **Terraform Change Safeguards** _\(planned\)_: We are evaluating stricter CI/CD policies and static analysis tools to detect unintended resource replacements before deployment. ### **Closing Note** We take this incident seriously and are committed to learning from it. We sincerely apologize for the disruption and thank you for your patience. If you have any questions or concerns, please reach out to us at [[email protected]](mailto:[email protected]).

Read the full incident report →

Started Apr 01, 2025, 01:34 PM UTC· resolved Apr 01, 2025, 09:05 PM UTC

1. identified Apr 01, 2025, 01:34 PM UTC

   Our web portal is experiencing elevated error rates (500 responses) and we're working to address and fully restore service.

2. resolved Apr 01, 2025, 02:42 PM UTC

   The incident has been resolved and service fully restored.

Read the full incident report →