SiteSpect experienced a minor incident on March 25, 2020 affecting Forte Cloud, lasting 2h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 25, 2020, 04:39 PM UTC
At approximately 16:15 UTC, the SiteSpect Engineering team was alerted to a potential issue and promptly started an investigation. In effort to prevent any service degradation, we are initiating bypass for all clients. We will update the status page as our investigation progresses.
- identified Mar 25, 2020, 06:07 PM UTC
The SiteSpect Engineering team has identified the issue and are currently working on a solution. We will continue to keep you updated.
- resolved Mar 25, 2020, 07:18 PM UTC
SiteSpect has implemented the solution. Service was restored at approximately 18:30 UTC.
- postmortem Mar 25, 2020, 09:14 PM UTC
**SiteSpect Cloud Incident RCA: 25 March 2020** At 16:07 UTC, SiteSpect Engineering was alerted to an issue involving our DNS service. At approximately 16:40 UTC, SiteSpect decided to initiate a cloud wide bypass to mitigate any potential service degradation. At 18:00 UTC, SiteSpect implemented a solution in our staging environments. At approximately 18:30 UTC, the solution was implemented in production and SiteSpect service was restored for all SiteSpect Cloud customers. The investigation uncovered that the third party service used by DNS "bind" software \([dlv.isc.org](http://dlv.isc.org)\) had inadvertently removed their own DNS A record, which prevented the SiteSpect platform from resolving origin hostnames. Besides SiteSpect, this impacted many other organizations worldwide. SiteSpect implemented a workaround to resolve the issue specifically for SiteSpect Cloud. The SiteSpect solution was implemented prior to ISC issuing a correction for the issue around 21:30 UTC. Despite the issue being fixed by ISC, SiteSpect has decided to discontinue use of the affected feature set. This will not affect the service level or how Customers are able to use the platform.