Chameleon experienced a major incident on June 10, 2025 affecting User profile API and Sidebar API and 1 more component, lasting 15h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 10, 2025, 12:57 PM UTC
We've identified an issue stemming from this Heroku incident https://status.heroku.com/incidents/2822 (upstream of Chameleon) which is causing us to have limited access to the heroku platform. We use Heroku for auto-scaling, console access, and deployment. This incident is currently affecting the availability of the Chameleon Dashboard, Builder, and our ability to run End-user Experiences (Tours, Launchers, Microsurveys).
- identified Jun 10, 2025, 01:44 PM UTC
The issue stems directly back to the Heroku outage (https://status.heroku.com/incidents/2822). Our current deployment of servers is running and handling the traffic to an acceptable level but we're unable to make any changes to the deployment, the number of servers (via autoscaling). We're seeing more total percentage of successful requests to Chameleon and that is also trending in the right direction. Will update again when we have new information
- identified Jun 10, 2025, 02:58 PM UTC
We continue to track the Heroku outage but continue to see intermittent issues with loading Chameleon Dashboard and generally connecting to Chameleon backend applications
- resolved Jun 11, 2025, 04:10 AM UTC
We've now seen consistent performance and response times from our Heroku instances for the past 3 hours and are making the incident as resolved. We will monitor the system as a whole now that everything is catching back up and the jobs/work to be done in the background queues. Thank you for your patience