Alloy had 42 outages in the last 2 years totaling 69h 21m of downtime — averaging 1.7 incidents per month.
There were 42 Alloy outages since June 12, 2024 totaling 69h 21m of downtime. Each is summarised below — incident details, duration, and resolution information.
Starting at 9:17 AM ET, the homepage analytics dashboards were not loading as expected. All other dashboard pages and services remained fully operational. As of 11:47 AM ET, the system has recovered and is fully operational.
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
We're currently investigating a period of latency and intermittent API unavailability that occurred between 11 and 11:15 ET.
We believe the cause of the latency is resolved.
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
The impact is limited to a subset of test suite endpoints used by the dashboard, our production apis were not impacted.
The issue has been resolved.
Between March 4 at 11:00 PM ET and March 5 at 10:07 AM ET, we experienced an elevated rate of memory-related errors impacting evaluations running through JQ input attributes. As a result, some evaluations returned a null value for the input attribute instead of the correct value, potentially interrupting decisioning in the Workflow. Impacted applications would need to be re-run. Only workflows using input attributes with JQ were impacted; other Workflow logic was not impacted. The issue has been identified and resolved, and all systems are now operating normally.
We recently identified and resolved an issue affecting some plugins initialized through the Alloy SDK. For a subset of SDK Plugins, applications initializing the SDK may have experienced unexpected behavior, which could present as UI instability or premature close events. We mitigated the issue by rolling back a recent SDK change. Following the rollback, normal plugin behavior was restored. If you continue to experience any issues, please contact Alloy Support.
We are currently investigating reports of elevated response times that affect both the Alloy Dashboard and API. Some customers may experience slower load times for the dashboard and longer response times for the API. Our engineering team is actively investigating.
The root cause has been identified. Response times for the Dashboard and API have returned to normal levels since 3:46 PM ET.
Between 11:20 AM and 12:04 PM ET, some workflows returned a 500 error when attempting to save a new version. Existing workflow versions and decisioning on existing workflow versions were not impacted. Once the root cause was identified as a previous code change, the issue was resolved at 12:04 PM ET when the affected change was rolled back, and all functionality related to workflow version saving was fully restored.
We are currently investigating an issue where Journey Analytics pages, Investigation detail pages, Fraud Attack Radar, and some Evaluation detail pages may not be loading data as expected. The issue is intermittent, and not all pages are impacted. Decisioning is not impacted. Journey Applications and Evaluations are completing successfully.
Our team continues to investigate the issue. At this time, there is no new information to share, but we are actively working to identify the root cause and will share updates as soon as possible.
We are still investigating this issue and will provide another update as soon as more information is available.
A fix has been implemented and all systems are operational.
Our monitoring has detected that calls to the OpenCorporates data integration are failing since Dec 2nd 4:52 PM ET. Requests are returning a 403 “Rate limit exceeded” error. We’re currently investigating and working with urgency with the OpenCorporates team to troubleshoot.
The OpenCorporates team has assisted in resolving the issue, and the OpenCorporates data service is now fully operational. Any applications that previously failed due to this incident can now be re-run.
A widespread Cloudflare outage is impacting multiple services, including Iovation and Middesk. We are monitoring the situation and investigating possible mitigation.
Our monitoring has detected that the services Middesk, Iovation, Iovation Registration, and Zenwork Compliancely TIN Match are seeing full downtime due to a CloudFlare outage. We have reached out to the services for more information, and we are awaiting resolution from CloudFlare. We will periodically update the status page as we learn more. More information about the CloudFlare outage can be found here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
Alloy is observing error rates have decreased across all impacted services as of 9:36 AM ET. CloudFlare has confirmed a fix has been implemented as of 9:42 AM ET. More information about the CloudFlare outage can be found here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
Our monitoring has detected that calls to Experian are failing
Our monitoring confirms that processing has returned to normal. We have not observed any further errors since 12:20 PM ET. Any applications that failed during this period can now be re-run successfully.
Our monitoring has detected that calls to ComplyAdvantage are failing, requests are returning a "timeout of 19000ms exceeded" error. The ComplyAdvantage team has confirmed they're aware of the incident and are investigating. We’re awaiting resolution from the ComplyAdvantage team.
Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 6:02 PM ET. Any applications that failed during this period can now be re-run successfully.
Our monitoring has detected that calls to Iovation are failing, requests are returning a “Internal Error” response. We’re awaiting resolution from the Iovation team.
TransUnion has provided an update: they've identified an issue that could be intermittently impacting some API calls. Their engineers are actively working to mitigate the issue. We will keep you posted as more information becomes available.
Our monitoring confirms that processing has returned to normal. We have not observed any further errors since 7:01 PM ET. Any applications that failed during this period can now be re-run successfully.
Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the GBG team.
Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.
Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the Equifax team.
Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.
Our monitoring has detected that calls to this service are failing, requests are returning a “Gateway” error with a 5XX response. We’re awaiting resolution from the LexisNexis team.
Our monitoring confirms that processing has returned to normal. We have not observed any timeout errors since 4:46 PM ET. Any applications that failed during this period can now be re-run successfully.
We are currently investigating reports that some users are not receiving MFA codes via text message. This only impacts users that rely on MFA codes delivered by SMS. Our team is actively working with our SMS vendor to identify and resolve the issue. If you’re unable to log in because you aren't receiving an SMS code, please reach out to your account's administrator to reset your MFA settings so you can switch to using an Authenticator App (such as Google Authenticator, Authy, or Microsoft Authenticator) instead of SMS codes. We will provide updates here as we learn more.
Our SMS vendor has confirmed that around 11:43 AM ET, there were SMS delivery delays when sending messages To T-mobile Network in the USA. They are now observing recovery when sending SMS messages over T-mobile and we’ve received confirmed reports from customers as well.
We have detected that calls to Qualifile were failing from 12:01AM ET due to a Qualifile maintenance window. We are monitoring to determine when their system is back up. We have confirmed that Qualifile processing has returned to normal. We have not observed any timeout errors since 2:03 PM ET. Any applications that failed during this period can now be re-run successfully.
We’re investigating reports of login failures affecting only customers who limit Alloy dashboard access via Allowlisted IPs. All other services and login methods remain fully operational. Our team is actively investigating and we will share updates as they become available.
We’ve determined that the login issues are specifically related to connections routed through Zscaler. Customers utilizing Zscaler solutions for secure internet access and experiencing difficulties logging into Alloy are recommended to reach out directly to Zscaler Support for assistance.
As part of our vanity domain rollout, Alloy’s dashboard domain has been updated from app.alloy.co to app.alloy.com. Customers using Zscaler will need to update their configuration to allowlist the following domain: *.alloy.com. This will allow specific hostnames go through specific outbound IPs, which have already been allowlisted in the Alloy dashboard settings, instead of Zscaler’s public IPs for generic outbound traffic.
Customers who use a security vendor to allowlist domains for making outbound requests though specific IPs to Alloy must now allowlist the new *.alloy.com domain. Example vendors include (but are not limited to) Zscaler, Cloudflare, Optimum, and Charter Communications.
We are resolving this incident after affected customers confirmed that updating their allowlisted domain .*alloy.com in their security vendor restored dashboard access. The issue was limited to customers who control Alloy dashboard access via Allowlisted IPs and use a security vendor that routes traffic over dedicated IPs based on domain.
We have identified a spike in 403 authentication errors in the Alloy dashboard as of around 14:30 ET. We are currently investigating.
We’ve resolved the issue that was causing the 403 errors, and our systems have returned to normal operations. Thank you for your patience while we worked to address this.
We are investigating issues with Alloy's Integrations Service
The issue has been identified and a fix is being implemented.
This incident has been resolved.
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
An interprocess communication issue resulted in a period of increased API latency and downtime from 12:28 to 12:44 ET (16 minutes). We have remediated the underlying issue and are working on an RCA of the incident now. API issues during this time will appear as either increased latency and/or HTTP 206 response codes. All API requests are retryable and can be retried now.