Chameleon experienced a minor incident on June 15, 2023 affecting User profile API, lasting 3h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 15, 2023, 02:40 PM UTC
We're seeing elevated error rates and retries related to processing Events and updates to Company properties. We're investigating the source of the problem and will report back here with more information.
- identified Jun 15, 2023, 03:54 PM UTC
We've identified a problem in how Heroku handles DNS resolution from Private spaces and are in contact with the support team. A large percentage of our servers are connecting to our database properly but the rest are failing to resolve the cluster address down to the addresses of the cluster instances. We're working on a possible workaround at the moment
- monitoring Jun 15, 2023, 05:28 PM UTC
We're seeing timeouts on approximately 4% of requests coming into Chameleon with the url prefix of https://fast.chameleon.io/observe/ and are still working to get back to fully operational
- resolved Jun 15, 2023, 05:51 PM UTC
This incident is now resolved. Our key metrics are back to normal and all queued jobs have been processed.