Chameleon.io experienced a minor incident on June 14, 2023 affecting HelpBar Search API, lasting 7h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 14, 2023, 07:44 PM UTC
HelpBar search results are seeing elevated error rates as some of the calls into our search service are not correctly resolving the DNS on the remote host. We're working with the Heroku team to identify the cause of this issue and will report back when we know more. In the meantime, we're keeping track of the hosts that are not correctly resolving the DNS and rotating them out of the Heroku cluster.
- monitoring Jun 14, 2023, 09:30 PM UTC
We've identified the issue as isolated to resolving ip DNS via IPv6. Our engineers have deployed a temporary fix while Heroku works on the resolution of this incident. As of now they have not posted on their status page (https://status.heroku.com/).
- resolved Jun 15, 2023, 03:05 AM UTC
We're marking this as resolved because the HelpBar search and DNS lookup are working as expected but we will plan to remove the temporary fix when Heroku gets to the bottom of the networking issue.