OAuth authentication issues
Timeline · 1 update
- resolved Mar 17, 2026, 12:47 PM UTC
OAuth authentication was not working for approximately 3pm est Sunday 2026-03-14 to 8:30 am est Tuesday 2026-03-16
Chameleon.io had 10 outages in the last 2 years totaling 42h 7m of downtime — averaging 0.4 incidents per month.
There were 10 Chameleon.io outages since June 10, 2025 totaling 42h 7m of downtime. Each is summarised below — incident details, duration, and resolution information.
OAuth authentication was not working for approximately 3pm est Sunday 2026-03-14 to 8:30 am est Tuesday 2026-03-16
We're looking into an issue where new magic links / OAuth / SSO are not working as expected
The issue has been identified and a fix is being implemented. **Note: If you're already logged in, you will continue to stay logged in and we will likely resolve this before you are automatically logged out**
This incident has been resolved.
Chameleon is not and was not not affected by the recently disclosed MongoDB vulnerability CVE-2025-14847 (aka. MongoBleed). It allows attackers to read arbitrary data from the database's heap memory. Chameleon's MongoDB instances are not publicly accessible and are also hosted with MongoDB Atlas which means they were patched before disclosure. We will, however, keep up to date on any developments from this incident
Chameleon **was not part of** the security incident reported by Mixpanel today (https://mixpanel.com/blog/sms-security-incident/). We will, however, keep up to date on any developments from this incident
We're seeing higher than normal error rates when editing within Chameleon. Our engineers have identified the cause and are working on a fix
Error rates are back down to normal levels
This incident has been resolved.
As of approximately 6:45 AM ET, Cloudflare reported an incident affecting many sites and vendors across the internet (Check https://www.cloudflarestatus.com/ for any direct updates). While Chameleon does not directly rely on Cloudflare, some services we depend on do. Consequently, you may experience the following: Copilot: OpenAI powers this feature and is currently impacted. Integrations: Data from integrations (Mixpanel, Amplitude, HubSpot, etc.) may be delayed until the incident is resolved. Recommendation: If your own product is experiencing downstream issues, we encourage you to use Chameleon to publish a downtime banner to alert your users. Please reach out if you need assistance setting this up.
According Cloudflare and others, this incident is resolved
During the AWS outage, Chameleon remained online and was not directly affected by the ongoing issues. All Chameleon Experiences were published on schedule, and new experiences went live as expected. Many customers also used Chameleon to notify *their customers* via in-product banners linking to their own status pages. During this period, we ingested and queued a large volume of data — typical for EU and US Eastern morning hours. However, today our auto-scaling was unable to bring additional resources online due to the AWS and Heroku issues. As a result, we are currently processing a backlog of data and background jobs. The current processing delay is approximately 30 minutes (Experience metrics are delayed by this amount, but microsurvey responses are not). This backlog will decrease quickly once auto-scaling resumes normal operation.
Updating to have a "Minor" impact
Our current application / background servers are running as-is but our ability to scale our cluster is impacted. We will need additional resources provisioned within the next few hours to bring the backlog back into a manageable length. We will continue to monitor and implement any fixes that we can along the way and will plan to send another update within the next 2-3 hours
Currently the Segment page (https://app.chameleon.io/segments) is only loading for a small percentage of requests due to upstream impact at Clickhouse => https://status.clickhouse.com
According to AWS "AWS services which rely on EC2 instance launches such as [Heroku] are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours" <= for Chameleon this means out data backlog should start processing more quickly automatically over the same time frame
The backlog of events has been processed and any new data coming in will be processed in near real time. TL;DR; Aside from the data backlog (peaked at 2 hours delay), and a ~10% error rate for loading Segments of users (in the Chameleon dashboard; no impact for your end-users), it was a relatively normal day at Chameleon in terms of the "rate of experiences delivered" for the baseline. Chameleon delivered approx. 200k downtime notices I hope today was not too stressful for you / your business and as always feel free to reach out with questions
Privacy update (not an incident) We’ve recently added a small number of trusted vendors to support key workflows across our platform, including: - Anthropic – AI functionality for workflow automations - Fathom – Call recording and transcription - Linear – Product issue and roadmap management - Synder – Financial system integrations - Mutiny – Website content personalization - Chili Piper – Meeting scheduling - Unify GTM – Go-to-market orchestration We’ve also streamlined our list by removing subprocessors no longer in use. These changes reflect our commitment to using modern, secure, and effective tools that help us improve our processes and deliver the best product and service to our customers. You can view the full, current list of subprocessors at our new subprocessors page: 👉 https://www.chameleon.io/security/subprocessors If you have any questions, feel free to reach out to us at [email protected]. Thanks, The Chameleon Team
We've identified instability in one of our internal APIs, which is causing intermittent 500 errors when editing and displaying Experiences. We've identified the root cause, and our team is actively working on stabilizing our services.
Solved: We've fixed the instability issues by scaling our service capacity.
We've identified an issue stemming from this Heroku incident https://status.heroku.com/incidents/2822 (upstream of Chameleon) which is causing us to have limited access to the heroku platform. We use Heroku for auto-scaling, console access, and deployment. This incident is currently affecting the availability of the Chameleon Dashboard, Builder, and our ability to run End-user Experiences (Tours, Launchers, Microsurveys).
The issue stems directly back to the Heroku outage (https://status.heroku.com/incidents/2822). Our current deployment of servers is running and handling the traffic to an acceptable level but we're unable to make any changes to the deployment, the number of servers (via autoscaling). We're seeing more total percentage of successful requests to Chameleon and that is also trending in the right direction. Will update again when we have new information
We continue to track the Heroku outage but continue to see intermittent issues with loading Chameleon Dashboard and generally connecting to Chameleon backend applications
We've now seen consistent performance and response times from our Heroku instances for the past 3 hours and are making the incident as resolved. We will monitor the system as a whole now that everything is catching back up and the jobs/work to be done in the background queues. Thank you for your patience