Zephyr Scale incident
Zephyr Scale is not accessible when using Firefox
Zephyr Scale experienced a major incident on October 20, 2021 affecting Zephyr Cloud (US), lasting 2h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 20, 2021, 01:11 PM UTC
We identified an issue with a change possibly related to HTTPS certificates that are blocking some users from accessing Zephyr Scale when using Firefox. If you have been affected by this issue, please use Google Chrome as a temporary workaround so that your work is not interrupted. We'll post updates very soon.
- monitoring Oct 20, 2021, 01:31 PM UTC
The root cause has been identified. The error was caused by a change that caused browsers to point to wrong IP addresses if DNS cache was enabled. Google Chrome wasn't affected. Firefox should be back to normal in a few minutes when an hour has passed from the time the problem was detected. If you are using Firefox and are still affected by this problem, please clear your DNS cache using the instructions below: 1) Type in your Firefox browser URL bar: about:networking#dns 2) Select "Clear DNS cache"
- resolved Oct 20, 2021, 03:17 PM UTC
The system has been operational again over the last ~2h.
- postmortem Oct 20, 2021, 03:18 PM UTC
Due to DNS changes, users using certain browsers with DNS caching features could not load the application anymore. This issue affected existing Zephyr Scale users who were using Firefox with DNS caching enabled. Firefox was the only browser identified to present this issue and its standard configuration for invalidating the DNS cache is one hour. After the Firefox cache has been invalidated, Zephyr Scale was again accessible to all users. The issue started around 1:30 pm BST and we got confirmation that it had been fixed about an hour later.