Alloy Outage History

Started Oct 20, 2025, 07:15 AM UTC· resolved Oct 21, 2025, 03:52 AM UTC

1. investigating Oct 20, 2025, 07:15 AM UTC

   Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

2. identified Oct 20, 2025, 07:43 AM UTC

   We have identified an issue with our cloud provider and are waiting for additional information

3. monitoring Oct 20, 2025, 09:48 AM UTC

   Our cloud provider confirmed there were service outages and have since released a fix. We are monitoring as systems recover.

4. investigating Oct 20, 2025, 11:09 AM UTC

   We are investigating ripple effects of the cloud outage on our outgoing webhooks and journeys processing.

5. investigating Oct 20, 2025, 12:51 PM UTC

   We are still working on resolving issues related to Webhooks processing. Customers may experience issues with Journey Application processing in this time

6. investigating Oct 20, 2025, 01:02 PM UTC

   Clients on /evaluations are not impacted. Clients implemented on Journeys may experience delayed journey API response and delayed webhooks that will self resolve. The is no data loss and no need for retries at this time

7. investigating Oct 20, 2025, 02:27 PM UTC

   We are still addressing the issues after the AWS outage. Alloy Engineering team is working on resolving the ongoing latency issues with Journeys and Webhooks. Any vendors that depend on AWS may also be experiencing degraded performance at the moment.

8. investigating Oct 20, 2025, 03:26 PM UTC

   We are still addressing the issues following the recent AWS outage. Customers may experience issues in the following areas: - API: We are investigating unavailability in the API. Requests may return 5XX errors. - Journeys and Webhooks: Ongoing latency as the Alloy Engineering team continues to work on resolution. - Logging into Alloy and Dashboard: Latency may occur while logging in, navigating the dashboard, or submitting reviews. Some users may intermittently see gateway timeout errors. - Data Vendor: Services that depend on AWS may also experience degraded performance, causing applications to fail and result in partial results. We appreciate your patience as our team continues to work toward full restoration of service.

9. investigating Oct 20, 2025, 04:02 PM UTC

   We are still addressing the issues following the recent AWS outage - you can follow their status at https://health.aws.amazon.com/health/status. Customers may experience issues in the following areas: - API: We are investigating intermittent unavailability in the API. Requests may return 5XX errors. - Journeys and Webhooks: Ongoing latency as the Alloy Engineering team continues to work on resolution. If there were any Journey Applications in a pending state at 11:29 AM ET they will not complete successfully, and any associated asynchronous tasks have been lost. If a Journey Application was run during this time and is not in a terminal state (e.g., Approved or Denied) - such as those waiting on step-ups or webhook actions - it will remain in an incomplete state. API calls for these applications will need to be re-run. - Logging into Alloy and Dashboard: Intermittent may occur while logging in, navigating the dashboard, or submitting reviews. Some users may intermittently see gateway timeout errors or blank screens. - Third party integrations: Services that depend on AWS may also experience degraded performance, causing applications to fail and result in partial results. Our engineering team is actively working to restore all services to full capacity as quickly as possible. We’ll continue to provide updates as more information becomes available

10. investigating Oct 20, 2025, 04:31 PM UTC

    We are still investigating the issue. AWS has implemented additional mitigation steps and is observing some recovery, but operations remain degraded and have not been fully restored. Alloy systems have not yet stabilized, and we are working closely with AWS support to restore full functionality.

11. investigating Oct 20, 2025, 05:07 PM UTC

    AWS continues to investigate ongoing EC2 and networking issues. Alloy services remain impacted. Our team remains in contact with AWS and will share updates every 30 minutes or as more substantial information becomes available.

12. investigating Oct 20, 2025, 05:37 PM UTC

    At this point, we unfortunately have no substantial update to share. We continue to investigate this with AWS at the highest priority. Thank you for the ongoing patience. We will keep you updated every 30 minutes or as more substantial information becomes available.

13. investigating Oct 20, 2025, 06:32 PM UTC

    AWS has reported progress in EC2 recovery, but Alloy’s full recovery remains dependent on AWS systems stabilizing. At this time, we are observing the following: - Webhooks: Backlog processing has begun, but a large queue remains. Real-time webhook delivery continues to experience delays. - Journeys: There continue to be intermittent Journeys failures. Some Journey Applications are failing to write to S3 - those will need to be retried once the incident is resolved. - Customer Dashboard: Intermittent latency may occur when loading or navigating the dashboard. - API: Intermittent latency persists for some requests. Our team continues to closely monitor AWS recovery and system performance. We’ll provide further updates as additional progress is made as close to 30 minute intervals as possible.

14. monitoring Oct 20, 2025, 07:02 PM UTC

    Network connectivity issues have been resolved. The Alloy Engineering team has rebuilt internal queues at 2:40 PM ET, and we are now seeing internal tests are now passing. - Webhooks: Backlog is still clearing. Realtime webhooks will continue to experience significant delays until the backlogged queue clears. - Dashboard: Has been restored to full functionality. - Journeys: Recovery is in progress. Journey Applications are starting to progress. - API: API latencies have resolved. We are continuing to monitor system performance as services come back online. We will be doing a root cause analysis into the full scope of impact. Further updates will be shared as recovery progresses.

15. monitoring Oct 20, 2025, 07:05 PM UTC

    We are continuing to monitor for any further issues.

16. investigating Oct 20, 2025, 07:51 PM UTC

    Our monitoring has detected an increase in automated test failures, and our engineering team is currently investigating.

17. investigating Oct 20, 2025, 08:21 PM UTC

    We are continuing to investigate connectivity issues within our cloud services. Customers may experience API timeouts while the issue persists. The dashboard remains fully operational. Webhooks are not impacted by the connectivity issues, but are still experiencing real-time processing delays as we continue clearing the backlog. We’ll provide further updates as more information becomes available.

18. investigating Oct 20, 2025, 08:32 PM UTC

    We are continuing to investigate connectivity issues within our cloud services. Customers may also experience Dashboard slowness while the issue persists. We’ll provide further updates as more information becomes available.

19. investigating Oct 20, 2025, 09:01 PM UTC

    At this point, we again unfortunately have no substantial update to share. We continue to investigate this with AWS at the highest priority.

20. investigating Oct 20, 2025, 09:57 PM UTC

    AWS continues to experience residual latency. All components are currently seeing elevated latency. Our engineering team is recycling service pods, and we are beginning to see signs of recovery. We are also investigating additional areas to further mitigate the impact. We will continue to closely monitor performance and share updates as they become available.

21. investigating Oct 20, 2025, 10:42 PM UTC

    We believe we have identified a third-party component that is throttling requests as part of recovery efforts to mitigate the ongoing impact from the AWS outage. This throttling is contributing to elevated network latency across Alloy services. We are optimistic that we can mitigate some of this impact with changes on our side and are actively working on those adjustments now. We will post another update at 8 PM ET or once substantial updates become available.

22. investigating Oct 21, 2025, 12:16 AM UTC

    Following a best-practice recommendation from our third-party vendor, we are deploying a change to our staging environment for validation and testing. We’ll proceed to production once we confirm the fix is effective. Next update to come after we complete the staging release and confirm results - we’ll aim for an update in about 30 minutes. Additionally, the webhook backlog continues to clear. Real-time processing of webhooks remains delayed until the queue is fully processed.

23. monitoring Oct 21, 2025, 12:46 AM UTC

    We’ve completed a release to production after confirming that all errors were resolved in staging. The production environment is now showing signs of recovery. We are actively monitoring progress. Additionally, we are still waiting for the webhook backlog to clear. Real-time processing of webhooks remains delayed until the queue is fully processed.

24. monitoring Oct 21, 2025, 01:39 AM UTC

    We’re pleased to share that normal operations have been restored for all major components. System latencies and performance metrics have returned to expected levels. All internal tests are passing, and we are no longer seeing errors related to the third-party component. Our teams work closely with AWS and the third-party vendor to conduct a comprehensive post mortem of the incident. We will share a Root Cause Analysis report once this review is complete. Please email [email protected] if you would like the RCA once it’s available. Please note that real-time processing of Webhooks is still delayed. Until our queue clears there will be delayed Webhooks. The queue is at about 60% cleared. We appreciate your patience and understanding throughout this event.

25. resolved Oct 21, 2025, 03:52 AM UTC

    As of 11:44 PM ET, the webhook backlog has been fully processed, and real-time webhook processing has been restored. All major Alloy services are now fully operational, with performance metrics and latency levels remaining stable. Our teams continue to work closely with AWS and the third-party vendor to complete a comprehensive post-mortem. We will share a Root Cause Analysis (RCA) once that review is finalized. If you’d like to receive the RCA once it's available, please email [email protected] . We appreciate your patience and understanding throughout this incident.

Read the full incident report →

Started Oct 17, 2025, 10:07 PM UTC· resolved Oct 17, 2025, 10:19 PM UTC

1. investigating Oct 17, 2025, 10:07 PM UTC

   Our monitoring has detected that calls to ComplyAdvantage are failing, requests are returning a "timeout of 19000ms exceeded" error. The ComplyAdvantage team has confirmed they're aware of the incident and are investigating. We’re awaiting resolution from the ComplyAdvantage team.

2. resolved Oct 17, 2025, 10:19 PM UTC

   Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 6:02 PM ET. Any applications that failed during this period can now be re-run successfully.

Read the full incident report →

Started Oct 17, 2025, 07:44 PM UTC· resolved Oct 17, 2025, 11:09 PM UTC

1. investigating Oct 17, 2025, 07:44 PM UTC

   Our monitoring has detected that calls to Iovation are failing, requests are returning a “Internal Error” response. We’re awaiting resolution from the Iovation team.

2. identified Oct 17, 2025, 08:20 PM UTC

   TransUnion has provided an update: they've identified an issue that could be intermittently impacting some API calls. Their engineers are actively working to mitigate the issue. We will keep you posted as more information becomes available.

3. resolved Oct 17, 2025, 11:09 PM UTC

   Our monitoring confirms that processing has returned to normal. We have not observed any further errors since 7:01 PM ET. Any applications that failed during this period can now be re-run successfully.

Read the full incident report →

Started Oct 08, 2025, 08:12 PM UTC· resolved Oct 08, 2025, 09:40 PM UTC

1. investigating Oct 08, 2025, 08:12 PM UTC

   Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the GBG team.

2. resolved Oct 08, 2025, 09:40 PM UTC

   Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.

Read the full incident report →

Started Oct 08, 2025, 08:12 PM UTC· resolved Oct 08, 2025, 09:41 PM UTC

1. investigating Oct 08, 2025, 08:12 PM UTC

   Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the Equifax team.

2. resolved Oct 08, 2025, 09:41 PM UTC

   Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.

Read the full incident report →

Started Oct 08, 2025, 08:11 PM UTC· resolved Oct 08, 2025, 09:41 PM UTC

1. investigating Oct 08, 2025, 08:11 PM UTC

   Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the LexisNexis team.

2. resolved Oct 08, 2025, 09:41 PM UTC

   Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.

Read the full incident report →

Started Sep 29, 2025, 05:40 PM UTC· resolved Sep 29, 2025, 06:16 PM UTC

1. investigating Sep 29, 2025, 05:40 PM UTC

   We are currently investigating reports that some users are not receiving MFA codes via text message. This only impacts users that rely on MFA codes delivered by SMS. Our team is actively working with our SMS vendor to identify and resolve the issue. If you’re unable to log in because you aren't receiving an SMS code, please reach out to your account's administrator to reset your MFA settings so you can switch to using an Authenticator App (such as Google Authenticator, Authy, or Microsoft Authenticator) instead of SMS codes. We will provide updates here as we learn more.

2. resolved Sep 29, 2025, 06:16 PM UTC

   Our SMS vendor has confirmed that around 11:43 AM ET, there were SMS delivery delays when sending messages To T-mobile Network in the USA. They are now observing recovery when sending SMS messages over T-mobile and we’ve received confirmed reports from customers as well.

Read the full incident report →

Started Sep 21, 2025, 04:00 AM UTC· resolved Sep 21, 2025, 04:00 AM UTC

1. resolved Sep 21, 2025, 07:01 AM UTC

   We have detected that calls to Qualifile were failing from 12:01AM ET due to a Qualifile maintenance window. We are monitoring to determine when their system is back up. We have confirmed that Qualifile processing has returned to normal. We have not observed any timeout errors since 2:03 PM ET. Any applications that failed during this period can now be re-run successfully.

Read the full incident report →

Started Sep 19, 2025, 06:13 PM UTC· resolved Sep 20, 2025, 04:32 PM UTC

1. investigating Sep 19, 2025, 06:13 PM UTC

   We’re investigating reports of login failures affecting only customers who limit Alloy dashboard access via Allowlisted IPs. All other services and login methods remain fully operational. Our team is actively investigating and we will share updates as they become available.

2. identified Sep 19, 2025, 06:52 PM UTC

   We’ve determined that the login issues are specifically related to connections routed through Zscaler. Customers utilizing Zscaler solutions for secure internet access and experiencing difficulties logging into Alloy are recommended to reach out directly to Zscaler Support for assistance.

3. identified Sep 19, 2025, 08:35 PM UTC

   As part of our vanity domain rollout, Alloy’s dashboard domain has been updated from app.alloy.co to app.alloy.com. Customers using Zscaler will need to update their configuration to allowlist the following domain: *.alloy.com. This will allow specific hostnames go through specific outbound IPs, which have already been allowlisted in the Alloy dashboard settings, instead of Zscaler’s public IPs for generic outbound traffic.

4. identified Sep 19, 2025, 10:47 PM UTC

   Customers who use a security vendor to allowlist domains for making outbound requests though specific IPs to Alloy must now allowlist the new *.alloy.com domain. Example vendors include (but are not limited to) Zscaler, Cloudflare, Optimum, and Charter Communications.

5. resolved Sep 20, 2025, 04:32 PM UTC

   We are resolving this incident after affected customers confirmed that updating their allowlisted domain .*alloy.com in their security vendor restored dashboard access. The issue was limited to customers who control Alloy dashboard access via Allowlisted IPs and use a security vendor that routes traffic over dedicated IPs based on domain.

Read the full incident report →

Started Sep 10, 2025, 06:55 PM UTC· resolved Sep 10, 2025, 07:49 PM UTC

1. investigating Sep 10, 2025, 06:55 PM UTC

   We have identified a spike in 403 authentication errors in the Alloy dashboard as of around 14:30 ET. We are currently investigating.

2. resolved Sep 10, 2025, 07:49 PM UTC

   We’ve resolved the issue that was causing the 403 errors, and our systems have returned to normal operations. Thank you for your patience while we worked to address this.

Read the full incident report →

Started Aug 13, 2025, 10:02 PM UTC· resolved Aug 13, 2025, 10:17 PM UTC

1. investigating Aug 13, 2025, 10:02 PM UTC

   We are investigating issues with Alloy's Integrations Service

2. identified Aug 13, 2025, 10:06 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Aug 13, 2025, 10:17 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 05, 2025, 04:37 PM UTC· resolved Aug 05, 2025, 04:48 PM UTC

1. investigating Aug 05, 2025, 04:37 PM UTC

   Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

2. investigating Aug 05, 2025, 04:44 PM UTC

   Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

3. resolved Aug 05, 2025, 04:48 PM UTC

   An interprocess communication issue resulted in a period of increased API latency and downtime from 12:28 to 12:44 ET (16 minutes). We have remediated the underlying issue and are working on an RCA of the incident now. API issues during this time will appear as either increased latency and/or HTTP 206 response codes. All API requests are retryable and can be retried now.

Read the full incident report →

Started Jul 24, 2025, 02:25 PM UTC· resolved Jul 24, 2025, 04:32 PM UTC

1. monitoring Jul 24, 2025, 02:25 PM UTC

   We’ve identified an issue causing some applications to fail to load with a ‘Something Went Wrong’ error. A fix has been deployed, and our engineering team is actively monitoring to ensure stability.

2. resolved Jul 24, 2025, 04:32 PM UTC

   The issue that caused some applications to fail to load and display a “Something Went Wrong” error has been fully resolved. Thank you for your patience during our investigation.

Read the full incident report →

Started Jul 18, 2025, 08:24 PM UTC· resolved Jul 24, 2025, 10:34 PM UTC

1. identified Jul 18, 2025, 08:24 PM UTC

   We’re currently experiencing a degraded performance for Fraud Attack Radar due to an upstream issue with our cloud provider. While fraud detection remains operational, some clients may experience a delay in alerts. Normal operations will be restored by end of day on July 28, 2025. Thank you for your understanding as we work to resolve this issue.

2. resolved Jul 24, 2025, 10:34 PM UTC

   The incident is now resolved, thank you for your patience.

Read the full incident report →

Started Jul 03, 2025, 05:12 PM UTC· resolved Jul 03, 2025, 05:39 PM UTC

1. investigating Jul 03, 2025, 05:12 PM UTC

   We’re currently investigating reports from some customers in the west coast regions experiencing blank page loads due to CORS-related errors. Our team is actively working to identify the root cause and will provide updates as more information becomes available.

2. resolved Jul 03, 2025, 05:39 PM UTC

   A fix has been released, and we’ve received confirmation from affected users that the issue is resolved. We’ll continue to monitor to ensure stability.

Read the full incident report →

Started Jun 16, 2025, 10:49 PM UTC· resolved Jun 17, 2025, 02:30 AM UTC

1. investigating Jun 16, 2025, 10:50 PM UTC

   We are currently investigating an issue where some applications remain in the review queue after being actioned and rerun evaluations are not appearing on the Evaluations page. Additionally, we are aware of a potential disruption where some customers are not seeing new evaluation records since approximately 4:40 PM CST. Our engineering team is actively investigating and we will provide updates as soon as more information becomes available.

2. identified Jun 17, 2025, 01:00 AM UTC

   Entities, Evaluations, Journey Applications, and Investigations that were created or updated since 5:45PM ET won't have up-to-date results when searched for in the dashboard. We have a fix in the works—verifying that it'll resolve the issue.

3. monitoring Jun 17, 2025, 02:03 AM UTC

   System is now recovering. We anticipate return to normal in 30min.

4. resolved Jun 17, 2025, 02:30 AM UTC

   The system has recovered.

Read the full incident report →

Started Jun 11, 2025, 06:37 PM UTC· resolved Jun 11, 2025, 07:09 PM UTC

1. investigating Jun 11, 2025, 07:02 PM UTC

   We have received reports of performance degradation with the Alloy dashboard. We are investigating this as a matter of priority.

2. investigating Jun 11, 2025, 07:02 PM UTC

   We are continuing to investigate this issue.

3. investigating Jun 11, 2025, 07:33 PM UTC

   We have identified and resolved the issue. Dashboard analytics page is operational.

4. resolved Jun 11, 2025, 07:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 09, 2025, 01:00 PM UTC· resolved Jun 09, 2025, 01:34 PM UTC

1. identified Jun 09, 2025, 01:15 PM UTC

   We have received reports of degradation with the Alloy SDK Veriff plugin. We are currently working on a fix.

2. identified Jun 09, 2025, 01:16 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Jun 09, 2025, 01:34 PM UTC

   Incident has been resolved.

Read the full incident report →

Started May 22, 2025, 03:18 PM UTC· resolved May 22, 2025, 03:25 PM UTC

1. investigating May 22, 2025, 03:18 PM UTC

   We’re currently investigating reports of elevated errors to the API. More updates to come shortly.

2. resolved May 22, 2025, 03:25 PM UTC

   A faulty API deployment caused timeout-related issues. The incident was identified at 11:01 AM and resolved by reverting the deployment at 11:14 AM.

Read the full incident report →