Vanilla Forums incident
Intermittent 522 errors from Cloudflare's DEN datacenter
Vanilla Forums experienced a minor incident on November 17, 2023 affecting VexxHost Canada Central Corporate Plan Communities, lasting 2d 16h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 17, 2023, 11:52 PM UTC
Since this morning at approximately 9:30, we've been observing intermittent 522 errors from Cloudflare that are isolated to their DEN (Denver, Colorado, USA) datacenter. No other datacenters are impacted. Users located in Colorado will experience intermittent 522 errors from Cloudflare. We have opened a ticket with Cloudflare and are awaiting updates.
- identified Nov 18, 2023, 02:51 AM UTC
We've had a long chat with Cloudflare's support team and the issue is being escalated to their network team. There appear to be some occasional packet drops between Denver and our Montréal datacenter. We estimate about 4% of traffic coming in from Denver is taking more than 5 seconds to respond and/or times out. Traffic originating from all other regions are unaffected and operating normally. We continue to investigate and await further responses from Cloudflare.
- resolved Nov 20, 2023, 07:17 AM UTC
Cloudflare has rerouted the traffic for that datacenter and the issue appear resolved. As of about 30 minutes ago, all traffic is back to normal and we're now seeing zero dropped requests from DEN.