Chameleon.io incident

Redis instance connection timeouts

Notice Resolved View vendor source →

Chameleon.io experienced a notice incident on February 6, 2024, lasting 1h 24m. The incident has been resolved; the full update timeline is below.

Started
Feb 06, 2024, 04:07 PM UTC
Resolved
Feb 06, 2024, 05:32 PM UTC
Duration
1h 24m
Detected by Pingoru
Feb 06, 2024, 04:07 PM UTC

Update timeline

  1. investigating Feb 06, 2024, 03:43 PM UTC

    We are currently investigating this issue.

  2. monitoring Feb 06, 2024, 04:07 PM UTC

    Our main redis instance had a failover to its secondary and is catching back up with traffic. This redis instance handles background processing and will take some time to catch back up with realtime. Customers may experience issues opening Chameleon's Builder in their applications, exporting reports, or loading certain Chameleon Dashboard pages.

  3. resolved Feb 06, 2024, 05:32 PM UTC

    This incident has been resolved. The analytics data received for the time between Feb 6th, 15:31 UTC and Feb 6th, 16:28 UTC were not processed in real time. Our team in the process of re-running any data received during this time frame including experience analytics. Additionally many background jobs during this will also be re-run including data exports. If you do not receive your data export by the end of the day today please re-export the data or reach out to support