Chameleon.io had 14 outages in the last 2 years totaling 76h 56m of downtime — averaging 0.6 incidents per month.
There were 14 Chameleon.io outages since January 30, 2025 totaling 76h 56m of downtime. Each is summarised below — incident details, duration, and resolution information.
We identified a backed-up queue that caused some requests to our backend to be rejected (with a 500/503 error code) -- the impact was that during the time period from 6:10 eastern to 10am eastern only approximately 92% of calls to chmln.identify were successful
OAuth authentication was not working for approximately 3pm est Sunday 2026-03-14 to 8:30 am est Tuesday 2026-03-16
We're looking into an issue where new magic links / OAuth / SSO are not working as expected
The issue has been identified and a fix is being implemented. **Note: If you're already logged in, you will continue to stay logged in and we will likely resolve this before you are automatically logged out**
This incident has been resolved.
Chameleon is not and was not not affected by the recently disclosed MongoDB vulnerability CVE-2025-14847 (aka. MongoBleed). It allows attackers to read arbitrary data from the database's heap memory. Chameleon's MongoDB instances are not publicly accessible and are also hosted with MongoDB Atlas which means they were patched before disclosure. We will, however, keep up to date on any developments from this incident
Chameleon **was not part of** the security incident reported by Mixpanel today (https://mixpanel.com/blog/sms-security-incident/). We will, however, keep up to date on any developments from this incident
We're seeing higher than normal error rates when editing within Chameleon. Our engineers have identified the cause and are working on a fix
Error rates are back down to normal levels
This incident has been resolved.
As of approximately 6:45 AM ET, Cloudflare reported an incident affecting many sites and vendors across the internet (Check https://www.cloudflarestatus.com/ for any direct updates). While Chameleon does not directly rely on Cloudflare, some services we depend on do. Consequently, you may experience the following: Copilot: OpenAI powers this feature and is currently impacted. Integrations: Data from integrations (Mixpanel, Amplitude, HubSpot, etc.) may be delayed until the incident is resolved. Recommendation: If your own product is experiencing downstream issues, we encourage you to use Chameleon to publish a downtime banner to alert your users. Please reach out if you need assistance setting this up.
According Cloudflare and others, this incident is resolved
During the AWS outage, Chameleon remained online and was not directly affected by the ongoing issues. All Chameleon Experiences were published on schedule, and new experiences went live as expected. Many customers also used Chameleon to notify *their customers* via in-product banners linking to their own status pages. During this period, we ingested and queued a large volume of data — typical for EU and US Eastern morning hours. However, today our auto-scaling was unable to bring additional resources online due to the AWS and Heroku issues. As a result, we are currently processing a backlog of data and background jobs. The current processing delay is approximately 30 minutes (Experience metrics are delayed by this amount, but microsurvey responses are not). This backlog will decrease quickly once auto-scaling resumes normal operation.
Updating to have a "Minor" impact
Our current application / background servers are running as-is but our ability to scale our cluster is impacted. We will need additional resources provisioned within the next few hours to bring the backlog back into a manageable length. We will continue to monitor and implement any fixes that we can along the way and will plan to send another update within the next 2-3 hours
Currently the Segment page (https://app.chameleon.io/segments) is only loading for a small percentage of requests due to upstream impact at Clickhouse => https://status.clickhouse.com
According to AWS "AWS services which rely on EC2 instance launches such as [Heroku] are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours" <= for Chameleon this means out data backlog should start processing more quickly automatically over the same time frame
The backlog of events has been processed and any new data coming in will be processed in near real time. TL;DR; Aside from the data backlog (peaked at 2 hours delay), and a ~10% error rate for loading Segments of users (in the Chameleon dashboard; no impact for your end-users), it was a relatively normal day at Chameleon in terms of the "rate of experiences delivered" for the baseline. Chameleon delivered approx. 200k downtime notices I hope today was not too stressful for you / your business and as always feel free to reach out with questions
Privacy update (not an incident) We’ve recently added a small number of trusted vendors to support key workflows across our platform, including: - Anthropic – AI functionality for workflow automations - Fathom – Call recording and transcription - Linear – Product issue and roadmap management - Synder – Financial system integrations - Mutiny – Website content personalization - Chili Piper – Meeting scheduling - Unify GTM – Go-to-market orchestration We’ve also streamlined our list by removing subprocessors no longer in use. These changes reflect our commitment to using modern, secure, and effective tools that help us improve our processes and deliver the best product and service to our customers. You can view the full, current list of subprocessors at our new subprocessors page: 👉 https://www.chameleon.io/security/subprocessors If you have any questions, feel free to reach out to us at [email protected]. Thanks, The Chameleon Team
We've identified instability in one of our internal APIs, which is causing intermittent 500 errors when editing and displaying Experiences. We've identified the root cause, and our team is actively working on stabilizing our services.
Solved: We've fixed the instability issues by scaling our service capacity.
We've identified an issue stemming from this Heroku incident https://status.heroku.com/incidents/2822 (upstream of Chameleon) which is causing us to have limited access to the heroku platform. We use Heroku for auto-scaling, console access, and deployment. This incident is currently affecting the availability of the Chameleon Dashboard, Builder, and our ability to run End-user Experiences (Tours, Launchers, Microsurveys).
The issue stems directly back to the Heroku outage (https://status.heroku.com/incidents/2822). Our current deployment of servers is running and handling the traffic to an acceptable level but we're unable to make any changes to the deployment, the number of servers (via autoscaling). We're seeing more total percentage of successful requests to Chameleon and that is also trending in the right direction. Will update again when we have new information
We continue to track the Heroku outage but continue to see intermittent issues with loading Chameleon Dashboard and generally connecting to Chameleon backend applications
We've now seen consistent performance and response times from our Heroku instances for the past 3 hours and are making the incident as resolved. We will monitor the system as a whole now that everything is catching back up and the jobs/work to be done in the background queues. Thank you for your patience
At Apr 16, 2025 12:01pm UTC, Chameleon deployed a code update that caused some Launchers to be shown to users not Segmented to them. Chameleon added an event listener to check for changes to the URL rules, but in case of a new match, this listener accounted only for URL rules, and didn't check Segmentation. The change was rolled back when discovered, at Apr 16, 2025 7:07pm UTC.
Chameleon continues to experience extremely high load and is taking steps to increase capacity above normal limits to address this. The current impact is that some requests to Chameleon are queued for a long time and/or timing out. In the UI this looks like page not loading, auto-saving not working, and for end-users experiences may not display. This currently appears to be happening for a < 10% of traffic but we'll continue to monitor as we work to resolve the issue
Our new capacity has been online for several hours now and is performing as expected. We will keep this incident open for another 24 hours in case of any new developments or issues. Thanks for out patience
We are currently investigating slow downs and 503 errors on profile requests.
We are continuing to monitor system performance as response times and error rates have improved over the past hour. No new end-user-facing timeouts have been observed recently, but we are still actively investigating to ensure stability.
This incident has been resolved, and all systems are operating as expected.
We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors
We are continuing to investigate this issue.
We have identified the following issues: 1) Experiences sometimes not being delivered to end-users. 2) Experiences sometimes not being able to be opened/edited in Chameleon Builder. We're actively monitoring and investigating the root cause.
This incident has been resolved and systems are back to normal.