Alloy Outage History
Alloy is up right nowPingoru has recorded 53 incidents for Alloy: 29 in the last 12 months and 8 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 1 of 720 (0%) |
| May 2026 | 3 | 2 of 744 (0%) |
| June 2026 | 3 | 1 of 720 (0%) |
| July 2026 | 3 | 26 of 744 (3%) |
| August 2026 | 3 | 7 of 744 (1%) |
| September 2026 | 4 | 6 of 720 (1%) |
| October 2026 | 1 | 2 of 518 (0%) |
Incidents, newest first
Minor October 9, 2026 File exports fail to download 1h 52m Customer Dashboard
Started Oct 09, 2026, 11:15 AM UTC· resolved Oct 09, 2026, 01:08 PM UTC
- investigating Oct 09, 2026, 11:15 AM UTC
File exports in Alloy Dashboard fail to download. We are currently investigating this issue.
- identified Oct 09, 2026, 11:53 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Oct 09, 2026, 01:08 PM UTC
This incident has been resolved.
Major September 30, 2026 Investigating issue processing API messages 31m Production API
Started Sep 30, 2026, 02:03 PM UTC· resolved Sep 30, 2026, 02:34 PM UTC
- investigating Sep 30, 2026, 02:03 PM UTC
We are currently investigating this issue.
Major September 10, 2026 Socure DocV is failing to complete for multiple clients 4h 38m SDK
Started Sep 10, 2026, 07:58 PM UTC· resolved Sep 11, 2026, 12:37 AM UTC
- investigating Sep 10, 2026, 07:58 PM UTC
We are currently investigating the issue.
- resolved Sep 11, 2026, 12:37 AM UTC
The issue is currently resolved. Socure DocV is available again. RCA is upcoming.
Notice September 9, 2026 [Retroactive] Issues loading Workflows, Journeys, and Application Queues – Customer Dashboard
Started Sep 09, 2026, 03:49 PM UTC· resolved Sep 09, 2026, 03:49 PM UTC
- resolved Sep 09, 2026, 03:49 PM UTC
Between approximately 11:18 AM and 11:38 AM ET, some users experienced intermittent issues loading Workflows, Journeys, and Application Queues in the Alloy Dashboard. A fix has been rolled out and we have confirmed that the affected pages are loading normally.
Minor September 4, 2026 Degraded Export Functionality 44m
Started Sep 04, 2026, 07:10 PM UTC· resolved Sep 04, 2026, 07:54 PM UTC
- investigating Sep 04, 2026, 07:10 PM UTC
Exports functionality has experienced degraded performance throughout this week; we’re investigating the issue. We are currently investigating this issue.
- resolved Sep 04, 2026, 07:54 PM UTC
We’ve identified the issue, deployed a fix, and confirmed export processing performance has returned to normal levels.
Notice August 27, 2026 Threatmetrix integration downtime –
Started Aug 27, 2026, 04:45 PM UTC· resolved Aug 27, 2026, 04:45 PM UTC
- resolved Aug 27, 2026, 10:20 PM UTC
Between 12:45 PM and 1:20 PM ET on August 27, a third-party data provider Threatmetrix experienced a service disruption that caused intermittent connection errors on device intelligence checks. During this window, some evaluations that use impacted service completed as partial rather than returning a full provider response, and some requests saw elevated latency or connector timeouts. The provider has confirmed the underlying issue is resolved, and error rates returned to normal by 1:20 PM ET. Evaluations submitted outside this window were not affected.
Minor August 25, 2026 Degraded performance 1h 54m Production API, Customer Dashboard, Sandbox API
Started Aug 25, 2026, 06:45 PM UTC· resolved Aug 25, 2026, 08:39 PM UTC
- investigating Aug 25, 2026, 06:45 PM UTC
Our automation detected degraded performance of Dashboard, Production API and Sandbox API components. We are currently investigating this issue.
- monitoring Aug 25, 2026, 06:56 PM UTC
Performance have been restored, all components are operational.
- resolved Aug 25, 2026, 08:39 PM UTC
Issue have been fully resolved. Elevated error rates were observed on August 25th between 14:18 and 14:22 PM ET and between 14:29 and 14:35 PM ET
Major August 20, 2026 loqate down address formatter unavailable 5h 4m Customer Dashboard
Started Aug 20, 2026, 03:38 PM UTC· resolved Aug 20, 2026, 08:43 PM UTC
- identified Aug 20, 2026, 03:38 PM UTC
We've detected issues with our address formatting provider. Decisioning is continuing to operate, but raw provided address will be sent to downstream services, which may result in some false negatives. We are currently in contact with the vendor and will update as soon as we know more.
- resolved Aug 20, 2026, 08:43 PM UTC
This incident has been resolved.
Major July 21, 2026 Journeys API Errors 21h 47m Production API
Started Jul 21, 2026, 05:56 PM UTC· resolved Jul 22, 2026, 03:44 PM UTC
- investigating Jul 21, 2026, 05:56 PM UTC
Some customers are experiencing elevated error rates for our Journeys API. We are investigating the impact now, and working on remediation.
- investigating Jul 21, 2026, 06:24 PM UTC
We have pushed a fixed which has resolved majority of the issues, we are currently monitoring
- investigating Jul 21, 2026, 06:36 PM UTC
The issue has now been resolved, and our team is still currently monitoring. We've also confirmed that this case impacted our dashboard, and that related impact has been addressed as well.
- investigating Jul 21, 2026, 06:52 PM UTC
Live traffic has been fully restored and the underlying issue has been resolved. We are continuing to address residual latency affecting dashboard indexing for requests submitted during the incident period, and the team is actively working toward full recovery.
- resolved Jul 22, 2026, 03:44 PM UTC
The issue affecting the Journeys API has been fully resolved, and service has returned to normal. We have completed monitoring and confirmed that related dashboard functionality has also recovered as of July 21, 2026 at 5:11 PM ET. Thank you for your patience while we worked to restore service.
Major July 17, 2026 Login Issues and Application Queues Not Displaying 8m Customer Dashboard
Started Jul 17, 2026, 02:42 PM UTC· resolved Jul 17, 2026, 02:51 PM UTC
- investigating Jul 17, 2026, 02:42 PM UTC
We are currently investigating reports of login issues and Application Queues not displaying for some customers. Our team is actively working to identify the cause and will provide another update as soon as more information is available.
- resolved Jul 17, 2026, 02:51 PM UTC
We have confirmed that login functionality and Application Queues have returned to normal operation, and no further impact is being observed. Thank you for your patience while we worked to resolve this issue.
Minor July 14, 2026 Integration Downtime: LexisNexis 3h 37m Third Party Data Integrations
Started Jul 14, 2026, 03:31 PM UTC· resolved Jul 14, 2026, 07:09 PM UTC
- investigating Jul 14, 2026, 03:31 PM UTC
Our monitoring has detected elevated failures for LexisNexis services, and our engineering team is actively investigating. We will provide an update as soon as more information is available.
- identified Jul 14, 2026, 03:53 PM UTC
We have confirmed that the elevated error rates are due to an outage affecting LexisNexis services. The following services are currently confirmed to be impacted: InstantID WorldCompliance RiskView LexisNexis Fraud Intelligence 2 Accurint InstantIDQA EnhancedPersonSearch We have contacted LexisNexis and are awaiting an update on their estimated timeline for service restoration. We will continue to monitor the situation and provide updates as we receive additional information.
- monitoring Jul 14, 2026, 04:31 PM UTC
We are seeing error rates decrease across the impacted LexisNexis services. Additionally, the LexisNexis status page, which was previously inaccessible during the incident, is now available again - https://status.lexisnexisrisk.com/. We will continue to closely monitor performance and will provide another update once we have confirmed that error rates have fully returned to normal.
- monitoring Jul 14, 2026, 05:07 PM UTC
We are continuing to see recovery across the impacted LexisNexis services, with most services now operating normally. We are still observing elevated error rates for RiskView and InstantVerify International and are continuing to monitor those services closely. We will provide another update once we have confirmed that all services have fully recovered.
- resolved Jul 14, 2026, 07:09 PM UTC
We have seen recovery across the impacted LexisNexis services and error rates has returned to expected levels. This incident is now considered resolved. We will continue to work with the LexisNexis team to obtain a Root Cause Analysis (RCA). If you would like to receive a copy of the RCA when it becomes available, please contact [email protected].
Critical June 18, 2026 API Downtime 24m Customer Dashboard
Started Jun 18, 2026, 06:03 PM UTC· resolved Jun 18, 2026, 06:27 PM UTC
- investigating Jun 18, 2026, 06:03 PM UTC
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
- investigating Jun 18, 2026, 06:15 PM UTC
We are currently investigating a spike in 500 errors affecting dashboard loading for some users . Our team is actively working to identify the root cause and mitigate the issue. We also observed a brief increase in API errors during this period, but API performance is now operating as expected.
- monitoring Jun 18, 2026, 06:19 PM UTC
We are seeing recovery in dashboard performance with error rates returning toward normal levels. We are closely monitoring the platform to ensure stability is fully restored.
- resolved Jun 18, 2026, 06:27 PM UTC
After further montioring, we have not observed any new errors. All systems are operating normally.
Notice May 8, 2026 [Retroactive] Homepage Dashboard Not Loading –
Started May 08, 2026, 01:00 PM UTC· resolved May 08, 2026, 01:00 PM UTC
- resolved May 08, 2026, 04:12 PM UTC
Starting at 9:17 AM ET, the homepage analytics dashboards were not loading as expected. All other dashboard pages and services remained fully operational. As of 11:47 AM ET, the system has recovered and is fully operational.
Major May 8, 2026 Latency and intermittent API unavailaiblity 34m Production API, Customer Dashboard
Started May 08, 2026, 03:23 AM UTC· resolved May 08, 2026, 03:57 AM UTC
- investigating May 08, 2026, 03:23 AM UTC
We're currently investigating a period of latency and intermittent API unavailability that occurred between 11 and 11:15 ET.
- resolved May 08, 2026, 03:57 AM UTC
We believe the cause of the latency is resolved.
Minor April 23, 2026 Test Suite endpoints experiencing an increase in errors 1h 7m Customer Dashboard
Started Apr 23, 2026, 02:25 PM UTC· resolved Apr 23, 2026, 03:33 PM UTC
- investigating Apr 23, 2026, 02:25 PM UTC
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
- investigating Apr 23, 2026, 02:39 PM UTC
The impact is limited to a subset of test suite endpoints used by the dashboard, our production apis were not impacted.
- resolved Apr 23, 2026, 03:33 PM UTC
The issue has been resolved.
Notice March 5, 2026 SDK Initialization Issue Affecting Some SDK Plugins –
Started Mar 05, 2026, 04:00 PM UTC· resolved Mar 05, 2026, 04:00 PM UTC
- resolved Mar 05, 2026, 05:50 PM UTC
We recently identified and resolved an issue affecting some plugins initialized through the Alloy SDK. For a subset of SDK Plugins, applications initializing the SDK may have experienced unexpected behavior, which could present as UI instability or premature close events. We mitigated the issue by rolling back a recent SDK change. Following the rollback, normal plugin behavior was restored. If you continue to experience any issues, please contact Alloy Support.
Notice March 5, 2026 [Retroactive] Elevated JQ Evaluation Memory Errors –
Started Mar 05, 2026, 04:00 AM UTC· resolved Mar 05, 2026, 04:00 AM UTC
- resolved Mar 09, 2026, 03:24 PM UTC
Between March 4 at 11:00 PM ET and March 5 at 10:07 AM ET, we experienced an elevated rate of memory-related errors impacting evaluations running through JQ input attributes. As a result, some evaluations returned a null value for the input attribute instead of the correct value, potentially interrupting decisioning in the Workflow. Impacted applications would need to be re-run. Only workflows using input attributes with JQ were impacted; other Workflow logic was not impacted. The issue has been identified and resolved, and all systems are now operating normally.
Minor February 19, 2026 Webhook Processing Delays 4h 14m Webhooks
Started Feb 19, 2026, 08:49 PM UTC· resolved Feb 20, 2026, 01:03 AM UTC
- identified Feb 19, 2026, 08:49 PM UTC
We are currently experiencing delays in realtime webhook processing. The root cause has been identified and we are applying mitigation measures. We are now monitoring recovery as the processing queue returns to normal levels. Webhook events continue to be accepted; however, processing and delivery responses may be delayed until the existing backlog has been fully cleared.
- monitoring Feb 19, 2026, 11:03 PM UTC
Realtime webhook processing has been fully restored. All new webhook events are now completing in realtime as expected. During the incident, webhook events generated between 2:24 PM - 4:38 PM ET and 5:42 - 5:45 ET were not successfully processed and were permanently lost. Webhooks generated outside of this time window were not impacted. This issue affects applications that are stuck at a Journey step dependent on an Alloy webhook response to proceed (for example, action nodes), as well as applications that completed with a Manual Review outcome but are not reflecting back in core after review. Applications submitted during the affected window that are currently blocked due to a missing webhook dependency will need to be replayed. We recommend resubmitting those applications via API. Re-running an application will leverage cached data by default and will not trigger new calls to external data vendors. A thorough postmortem will be completed a Root Cause Analysis (RCA) can be shared with customers - please email [email protected] if you would like to receive the RCA. We are also investing in further improvements to the reliability and resiliency of our webhook queueing mechanism in the future to prevent event loss in the future.
- resolved Feb 20, 2026, 01:03 AM UTC
Realtime webhook processing has been fully restored, and has maintained stability after monitoring. All new webhook events are now completing in realtime as expected. To further strengthen system stability, the engineering team has deployed an additional mitigation to prevent webhook queue backlogging under similar spike conditions. A thorough postmortem will be completed a Root Cause Analysis (RCA) can be shared with customers - please email [email protected] if you would like to receive the RCA. We are also investing in further improvements to the reliability and resiliency of our webhook queueing mechanism in the future to prevent event loss in the future. Please note, action is required to ensure applications process: During the incident, webhook events generated between 2:24 PM - 4:38 PM ET and 5:42 - 5:45 ET were not successfully processed and were permanently lost. Webhooks generated outside of this time window were not impacted. This issue affects applications that are stuck at a Journey step dependent on an Alloy webhook response to proceed (for example, action nodes), as well as applications that completed with a Manual Review outcome but are not reflecting back in core after review. Applications submitted during the affected window that are currently blocked due to a missing webhook dependency will need to be replayed. We recommend resubmitting those applications via API. Re-running an application will leverage cached data by default and will not trigger new calls to external data vendors.
Minor February 17, 2026 Increased Response Times for Dashboard and API 11m Production API, Customer Dashboard
Started Feb 17, 2026, 09:03 PM UTC· resolved Feb 17, 2026, 09:14 PM UTC
- investigating Feb 17, 2026, 09:03 PM UTC
We are currently investigating reports of elevated response times that affect both the Alloy Dashboard and API. Some customers may experience slower load times for the dashboard and longer response times for the API. Our engineering team is actively investigating.
- resolved Feb 17, 2026, 09:14 PM UTC
The root cause has been identified. Response times for the Dashboard and API have returned to normal levels since 3:46 PM ET.
Notice February 2, 2026 [Retroactive] Errors When Saving New Workflow Versions – Customer Dashboard
Started Feb 02, 2026, 05:27 PM UTC· resolved Feb 02, 2026, 05:27 PM UTC
- resolved Feb 02, 2026, 05:27 PM UTC
Between 11:20 AM and 12:04 PM ET, some workflows returned a 500 error when attempting to save a new version. Existing workflow versions and decisioning on existing workflow versions were not impacted. Once the root cause was identified as a previous code change, the issue was resolved at 12:04 PM ET when the affected change was rolled back, and all functionality related to workflow version saving was fully restored.
Minor January 8, 2026 Some Analytics and Detail Pages Not Loading Data 2h 47m Customer Dashboard
Started Jan 08, 2026, 06:43 PM UTC· resolved Jan 08, 2026, 09:30 PM UTC
- investigating Jan 08, 2026, 06:43 PM UTC
We are currently investigating an issue where Journey Analytics pages, Investigation detail pages, Fraud Attack Radar, and some Evaluation detail pages may not be loading data as expected. The issue is intermittent, and not all pages are impacted. Decisioning is not impacted. Journey Applications and Evaluations are completing successfully.
- investigating Jan 08, 2026, 07:22 PM UTC
Our team continues to investigate the issue. At this time, there is no new information to share, but we are actively working to identify the root cause and will share updates as soon as possible.
- investigating Jan 08, 2026, 08:39 PM UTC
We are still investigating this issue and will provide another update as soon as more information is available.
- resolved Jan 08, 2026, 10:40 PM UTC
A fix has been implemented and all systems are operational.
Minor December 3, 2025 Integration Downtime: OpenCorporates 4h 14m Third Party Data Integrations
Started Dec 03, 2025, 06:37 PM UTC· resolved Dec 03, 2025, 10:51 PM UTC
- investigating Dec 03, 2025, 06:37 PM UTC
Our monitoring has detected that calls to the OpenCorporates data integration are failing since Dec 2nd 4:52 PM ET. Requests are returning a 403 “Rate limit exceeded” error. We’re currently investigating and working with urgency with the OpenCorporates team to troubleshoot.
- resolved Dec 03, 2025, 10:51 PM UTC
The OpenCorporates team has assisted in resolving the issue, and the OpenCorporates data service is now fully operational. Any applications that previously failed due to this incident can now be re-run.
Notice November 18, 2025 CloudFlare incident causing increased errors for data services 1h 30m Production API
Started Nov 18, 2025, 01:37 PM UTC· resolved Nov 18, 2025, 03:08 PM UTC
- investigating Nov 18, 2025, 01:37 PM UTC
A widespread Cloudflare outage is impacting multiple services, including Iovation and Middesk. We are monitoring the situation and investigating possible mitigation.
- investigating Nov 18, 2025, 01:52 PM UTC
Our monitoring has detected that the services Middesk, Iovation, Iovation Registration, and Zenwork Compliancely TIN Match are seeing full downtime due to a CloudFlare outage. We have reached out to the services for more information, and we are awaiting resolution from CloudFlare. We will periodically update the status page as we learn more. More information about the CloudFlare outage can be found here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
- resolved Nov 18, 2025, 03:08 PM UTC
Alloy is observing error rates have decreased across all impacted services as of 9:36 AM ET. CloudFlare has confirmed a fix has been implemented as of 9:42 AM ET. More information about the CloudFlare outage can be found here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
Minor November 3, 2025 Integration Downtime: Experian 13m Production API
Started Nov 03, 2025, 05:06 PM UTC· resolved Nov 03, 2025, 05:20 PM UTC
- investigating Nov 03, 2025, 05:06 PM UTC
Our monitoring has detected that calls to Experian are failing
- resolved Nov 03, 2025, 06:44 PM UTC
Our monitoring confirms that processing has returned to normal. We have not observed any further errors since 12:20 PM ET. Any applications that failed during this period can now be re-run successfully.
Critical October 20, 2025 Increased API Errors - Cloud Provider Issues 20h 36m Production API, Customer Dashboard, Webhooks
Started Oct 20, 2025, 07:15 AM UTC· resolved Oct 21, 2025, 03:52 AM UTC
- investigating Oct 20, 2025, 07:15 AM UTC
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
- identified Oct 20, 2025, 07:43 AM UTC
We have identified an issue with our cloud provider and are waiting for additional information
- monitoring Oct 20, 2025, 09:48 AM UTC
Our cloud provider confirmed there were service outages and have since released a fix. We are monitoring as systems recover.
- investigating Oct 20, 2025, 11:09 AM UTC
We are investigating ripple effects of the cloud outage on our outgoing webhooks and journeys processing.
- investigating Oct 20, 2025, 12:51 PM UTC
We are still working on resolving issues related to Webhooks processing. Customers may experience issues with Journey Application processing in this time
- investigating Oct 20, 2025, 01:02 PM UTC
Clients on /evaluations are not impacted. Clients implemented on Journeys may experience delayed journey API response and delayed webhooks that will self resolve. The is no data loss and no need for retries at this time
- investigating Oct 20, 2025, 02:27 PM UTC
We are still addressing the issues after the AWS outage. Alloy Engineering team is working on resolving the ongoing latency issues with Journeys and Webhooks. Any vendors that depend on AWS may also be experiencing degraded performance at the moment.
- investigating Oct 20, 2025, 03:26 PM UTC
We are still addressing the issues following the recent AWS outage. Customers may experience issues in the following areas: - API: We are investigating unavailability in the API. Requests may return 5XX errors. - Journeys and Webhooks: Ongoing latency as the Alloy Engineering team continues to work on resolution. - Logging into Alloy and Dashboard: Latency may occur while logging in, navigating the dashboard, or submitting reviews. Some users may intermittently see gateway timeout errors. - Data Vendor: Services that depend on AWS may also experience degraded performance, causing applications to fail and result in partial results. We appreciate your patience as our team continues to work toward full restoration of service.
- investigating Oct 20, 2025, 04:02 PM UTC
We are still addressing the issues following the recent AWS outage - you can follow their status at https://health.aws.amazon.com/health/status. Customers may experience issues in the following areas: - API: We are investigating intermittent unavailability in the API. Requests may return 5XX errors. - Journeys and Webhooks: Ongoing latency as the Alloy Engineering team continues to work on resolution. If there were any Journey Applications in a pending state at 11:29 AM ET they will not complete successfully, and any associated asynchronous tasks have been lost. If a Journey Application was run during this time and is not in a terminal state (e.g., Approved or Denied) - such as those waiting on step-ups or webhook actions - it will remain in an incomplete state. API calls for these applications will need to be re-run. - Logging into Alloy and Dashboard: Intermittent may occur while logging in, navigating the dashboard, or submitting reviews. Some users may intermittently see gateway timeout errors or blank screens. - Third party integrations: Services that depend on AWS may also experience degraded performance, causing applications to fail and result in partial results. Our engineering team is actively working to restore all services to full capacity as quickly as possible. We’ll continue to provide updates as more information becomes available
- investigating Oct 20, 2025, 04:31 PM UTC
We are still investigating the issue. AWS has implemented additional mitigation steps and is observing some recovery, but operations remain degraded and have not been fully restored. Alloy systems have not yet stabilized, and we are working closely with AWS support to restore full functionality.
- investigating Oct 20, 2025, 05:07 PM UTC
AWS continues to investigate ongoing EC2 and networking issues. Alloy services remain impacted. Our team remains in contact with AWS and will share updates every 30 minutes or as more substantial information becomes available.
- investigating Oct 20, 2025, 05:37 PM UTC
At this point, we unfortunately have no substantial update to share. We continue to investigate this with AWS at the highest priority. Thank you for the ongoing patience. We will keep you updated every 30 minutes or as more substantial information becomes available.
- investigating Oct 20, 2025, 06:32 PM UTC
AWS has reported progress in EC2 recovery, but Alloy’s full recovery remains dependent on AWS systems stabilizing. At this time, we are observing the following: - Webhooks: Backlog processing has begun, but a large queue remains. Real-time webhook delivery continues to experience delays. - Journeys: There continue to be intermittent Journeys failures. Some Journey Applications are failing to write to S3 - those will need to be retried once the incident is resolved. - Customer Dashboard: Intermittent latency may occur when loading or navigating the dashboard. - API: Intermittent latency persists for some requests. Our team continues to closely monitor AWS recovery and system performance. We’ll provide further updates as additional progress is made as close to 30 minute intervals as possible.
- monitoring Oct 20, 2025, 07:02 PM UTC
Network connectivity issues have been resolved. The Alloy Engineering team has rebuilt internal queues at 2:40 PM ET, and we are now seeing internal tests are now passing. - Webhooks: Backlog is still clearing. Realtime webhooks will continue to experience significant delays until the backlogged queue clears. - Dashboard: Has been restored to full functionality. - Journeys: Recovery is in progress. Journey Applications are starting to progress. - API: API latencies have resolved. We are continuing to monitor system performance as services come back online. We will be doing a root cause analysis into the full scope of impact. Further updates will be shared as recovery progresses.
- monitoring Oct 20, 2025, 07:05 PM UTC
We are continuing to monitor for any further issues.
- investigating Oct 20, 2025, 07:51 PM UTC
Our monitoring has detected an increase in automated test failures, and our engineering team is currently investigating.
- investigating Oct 20, 2025, 08:21 PM UTC
We are continuing to investigate connectivity issues within our cloud services. Customers may experience API timeouts while the issue persists. The dashboard remains fully operational. Webhooks are not impacted by the connectivity issues, but are still experiencing real-time processing delays as we continue clearing the backlog. We’ll provide further updates as more information becomes available.
- investigating Oct 20, 2025, 08:32 PM UTC
We are continuing to investigate connectivity issues within our cloud services. Customers may also experience Dashboard slowness while the issue persists. We’ll provide further updates as more information becomes available.
- investigating Oct 20, 2025, 09:01 PM UTC
At this point, we again unfortunately have no substantial update to share. We continue to investigate this with AWS at the highest priority.
- investigating Oct 20, 2025, 09:57 PM UTC
AWS continues to experience residual latency. All components are currently seeing elevated latency. Our engineering team is recycling service pods, and we are beginning to see signs of recovery. We are also investigating additional areas to further mitigate the impact. We will continue to closely monitor performance and share updates as they become available.
- investigating Oct 20, 2025, 10:42 PM UTC
We believe we have identified a third-party component that is throttling requests as part of recovery efforts to mitigate the ongoing impact from the AWS outage. This throttling is contributing to elevated network latency across Alloy services. We are optimistic that we can mitigate some of this impact with changes on our side and are actively working on those adjustments now. We will post another update at 8 PM ET or once substantial updates become available.
- investigating Oct 21, 2025, 12:16 AM UTC
Following a best-practice recommendation from our third-party vendor, we are deploying a change to our staging environment for validation and testing. We’ll proceed to production once we confirm the fix is effective. Next update to come after we complete the staging release and confirm results - we’ll aim for an update in about 30 minutes. Additionally, the webhook backlog continues to clear. Real-time processing of webhooks remains delayed until the queue is fully processed.
- monitoring Oct 21, 2025, 12:46 AM UTC
We’ve completed a release to production after confirming that all errors were resolved in staging. The production environment is now showing signs of recovery. We are actively monitoring progress. Additionally, we are still waiting for the webhook backlog to clear. Real-time processing of webhooks remains delayed until the queue is fully processed.
- monitoring Oct 21, 2025, 01:39 AM UTC
We’re pleased to share that normal operations have been restored for all major components. System latencies and performance metrics have returned to expected levels. All internal tests are passing, and we are no longer seeing errors related to the third-party component. Our teams work closely with AWS and the third-party vendor to conduct a comprehensive post mortem of the incident. We will share a Root Cause Analysis report once this review is complete. Please email [email protected] if you would like the RCA once it’s available. Please note that real-time processing of Webhooks is still delayed. Until our queue clears there will be delayed Webhooks. The queue is at about 60% cleared. We appreciate your patience and understanding throughout this event.
- resolved Oct 21, 2025, 03:52 AM UTC
As of 11:44 PM ET, the webhook backlog has been fully processed, and real-time webhook processing has been restored. All major Alloy services are now fully operational, with performance metrics and latency levels remaining stable. Our teams continue to work closely with AWS and the third-party vendor to complete a comprehensive post-mortem. We will share a Root Cause Analysis (RCA) once that review is finalized. If you’d like to receive the RCA once it's available, please email [email protected] . We appreciate your patience and understanding throughout this incident.