GraphCDN Outage History

## Summary On August 20, 2025, our services experienced elevated errors on our KV stores due to a third-party vendor issue. The incident was caused by a faulty enhancement deployment on the vendor's platform, which was subsequently rolled back to restore service. ## Root Cause Third-party vendor deployed a faulty enhancement that caused backend errors for a subset of customers, primarily in Singapore and California regions. ## Resolution Vendor engineering team rolled back the problematic deployment and confirmed service restoration. ## Lessons Learned * Vendor responded quickly with clear communication * Need better early detection of vendor-side issues * Consider additional monitoring for vendor dependencies ‌ Related: [https://www.fastlystatus.com/incident/377843](https://www.fastlystatus.com/incident/377843).

During a routine employee offboarding, we revoked that employee’s access to Fastly. Revoking their access to Fastly also revoked all access tokens that engineer created. Unfortunately, this included the central API token all our systems use to communicate with the Fastly API. This had two immediate impacts: 1. Purging started failing silently: Stellate’s purging API kept returning successful responses even though data would not be evicted from the cache. 2. Service configuration updates failing silently: Service configuration updates appeared to persist even though they were not updated in the CDN. As part of the incident response, we switched the central Fastly API token to a new token owned by a shared engineering account. Further, we will work on gaining better visibility and alerting on failure conditions with the purging API, as well as audit all tokens in use by our services to ensure they are not owned by individual engineers.