GraphCDN incident
Issues with GraphQL Metrics ingesting new data
GraphCDN experienced a major incident on August 29, 2024 affecting GraphQL Metrics, lasting 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 29, 2024, 09:10 AM UTC
We are currently looking into an issue with our GraphQL Metrics systems, which prevents new data to show up on the dashboard. All other systems are working as expected and our GraphQL Edge Caching and GraphQL Rate Limiting systems are not affected.
- identified Aug 29, 2024, 09:47 AM UTC
The issue has been identified as an issue with one of our infrastructure providers. and their ClickPipes service. They have opened an investigation and updated their status page at https://status.clickhouse.com/incidents/f6j8dfnyy6dn
- monitoring Aug 29, 2024, 10:13 AM UTC
We are seeing signs of recovery, and new data is again being ingested into our GraphQL Metrics systems. We are monitoring all systems and are in touch with our infrastructure provider for further updates. Please note that the GraphQL Metrics system might lag behind near real-time for a while as queued updates get processed.
- resolved Aug 29, 2024, 10:34 AM UTC
This issue has been resolved. We'll continue monitoring all services, but we don't expect any further issues.