Chameleon incident

Chameleon app / API slowdown

Notice Resolved View vendor source →

Chameleon experienced a notice incident on May 2, 2023 affecting User profile API and Sidebar API and 1 more component, lasting 4h 9m. The incident has been resolved; the full update timeline is below.

Started
May 02, 2023, 12:44 PM UTC
Resolved
May 02, 2023, 04:53 PM UTC
Duration
4h 9m
Detected by Pingoru
May 02, 2023, 12:44 PM UTC

Affected components

User profile APISidebar APIData API

Update timeline

  1. investigating May 02, 2023, 12:44 PM UTC

    We're investigating a slowdown in serving API requests to Chameleon

  2. identified May 02, 2023, 01:03 PM UTC

    app.chameleon.io continues to timeout when loading -- the underlying cause appears to be a recent maintenance event at our MongoDB instance which has left it only partially available and are working with their team toward resolution

  3. identified May 02, 2023, 01:33 PM UTC

    As an update to the ongoing issue we're experiencing, our database provider's team (MongoDB Atlas) has identified the underlying issue is actively working on the resolution. Chameleon, for the most part, will continue to be unavailable until this is resolved. The primary impact is that new Chameleon Experiences will not be displayed to your users and the Chameleon dashboard (app.chameleon.io) is not loading.

  4. monitoring May 02, 2023, 01:45 PM UTC

    This incident is now on it's way to resolution; from our testing and metrics the Chameleon dashboard (app.chameleon.io) is now loading and Experiences are being displayed (e.g. Launchers, Tours). We will provide a final update here when all of our metrics/signals have returned to normal.

  5. resolved May 02, 2023, 04:53 PM UTC

    While we're confident this incident is over for the immediate future, and while everything is catching back up (for a few types of data updates, mainly to companies) as everything returns to normal, we will be monitoring a few things more closely over the next few days.