OAuth authentication issues
Timeline · 1 update
- resolved Mar 17, 2026, 12:47 PM UTC
OAuth authentication was not working for approximately 3pm est Sunday 2026-03-14 to 8:30 am est Tuesday 2026-03-16
Chameleon had 9 outages in the last 2 years totaling 26h 54m of downtime — averaging 0.4 incidents per month.
There were 9 Chameleon outages since June 25, 2025 totaling 26h 54m of downtime. Each is summarised below — incident details, duration, and resolution information.
OAuth authentication was not working for approximately 3pm est Sunday 2026-03-14 to 8:30 am est Tuesday 2026-03-16
We're looking into an issue where new magic links / OAuth / SSO are not working as expected
The issue has been identified and a fix is being implemented. **Note: If you're already logged in, you will continue to stay logged in and we will likely resolve this before you are automatically logged out**
This incident has been resolved.
Chameleon is not and was not not affected by the recently disclosed MongoDB vulnerability CVE-2025-14847 (aka. MongoBleed). It allows attackers to read arbitrary data from the database's heap memory. Chameleon's MongoDB instances are not publicly accessible and are also hosted with MongoDB Atlas which means they were patched before disclosure. We will, however, keep up to date on any developments from this incident
Chameleon **was not part of** the security incident reported by Mixpanel today (https://mixpanel.com/blog/sms-security-incident/). We will, however, keep up to date on any developments from this incident
We're seeing higher than normal error rates when editing within Chameleon. Our engineers have identified the cause and are working on a fix
Error rates are back down to normal levels
This incident has been resolved.
As of approximately 6:45 AM ET, Cloudflare reported an incident affecting many sites and vendors across the internet (Check https://www.cloudflarestatus.com/ for any direct updates). While Chameleon does not directly rely on Cloudflare, some services we depend on do. Consequently, you may experience the following: Copilot: OpenAI powers this feature and is currently impacted. Integrations: Data from integrations (Mixpanel, Amplitude, HubSpot, etc.) may be delayed until the incident is resolved. Recommendation: If your own product is experiencing downstream issues, we encourage you to use Chameleon to publish a downtime banner to alert your users. Please reach out if you need assistance setting this up.
According Cloudflare and others, this incident is resolved
During the AWS outage, Chameleon remained online and was not directly affected by the ongoing issues. All Chameleon Experiences were published on schedule, and new experiences went live as expected. Many customers also used Chameleon to notify *their customers* via in-product banners linking to their own status pages. During this period, we ingested and queued a large volume of data — typical for EU and US Eastern morning hours. However, today our auto-scaling was unable to bring additional resources online due to the AWS and Heroku issues. As a result, we are currently processing a backlog of data and background jobs. The current processing delay is approximately 30 minutes (Experience metrics are delayed by this amount, but microsurvey responses are not). This backlog will decrease quickly once auto-scaling resumes normal operation.
Updating to have a "Minor" impact
Our current application / background servers are running as-is but our ability to scale our cluster is impacted. We will need additional resources provisioned within the next few hours to bring the backlog back into a manageable length. We will continue to monitor and implement any fixes that we can along the way and will plan to send another update within the next 2-3 hours
Currently the Segment page (https://app.chameleon.io/segments) is only loading for a small percentage of requests due to upstream impact at Clickhouse => https://status.clickhouse.com
According to AWS "AWS services which rely on EC2 instance launches such as [Heroku] are working through their backlog of EC2 instance launches successfully and we anticipate full recovery of the backlog over the next two hours" <= for Chameleon this means out data backlog should start processing more quickly automatically over the same time frame
The backlog of events has been processed and any new data coming in will be processed in near real time. TL;DR; Aside from the data backlog (peaked at 2 hours delay), and a ~10% error rate for loading Segments of users (in the Chameleon dashboard; no impact for your end-users), it was a relatively normal day at Chameleon in terms of the "rate of experiences delivered" for the baseline. Chameleon delivered approx. 200k downtime notices I hope today was not too stressful for you / your business and as always feel free to reach out with questions
Privacy update (not an incident) We’ve recently added a small number of trusted vendors to support key workflows across our platform, including: - Anthropic – AI functionality for workflow automations - Fathom – Call recording and transcription - Linear – Product issue and roadmap management - Synder – Financial system integrations - Mutiny – Website content personalization - Chili Piper – Meeting scheduling - Unify GTM – Go-to-market orchestration We’ve also streamlined our list by removing subprocessors no longer in use. These changes reflect our commitment to using modern, secure, and effective tools that help us improve our processes and deliver the best product and service to our customers. You can view the full, current list of subprocessors at our new subprocessors page: 👉 https://www.chameleon.io/security/subprocessors If you have any questions, feel free to reach out to us at [email protected]. Thanks, The Chameleon Team
We've identified instability in one of our internal APIs, which is causing intermittent 500 errors when editing and displaying Experiences. We've identified the root cause, and our team is actively working on stabilizing our services.
Solved: We've fixed the instability issues by scaling our service capacity.