Major outage due to infrastructure failure

Update timeline

1. investigating Nov 01, 2026, 02:15 PM UTC

   Investigating the cause of elevated error rates since 14:10 GMT

2. identified Nov 01, 2026, 02:42 PM UTC

   We are currently experiencing a service disruption caused by an outage affecting our Redis infrastructure. Our infrastructure provider has placed the affected Redis instance into maintenance mode, and we are currently unable to restore service. We are working with our provider to resolve the issue as quickly as possible and will share further updates as we learn more. We are sorry for the disruption.

3. monitoring Nov 01, 2026, 03:05 PM UTC

   Normal service resumed at approximately 14:40 UTC following the Redis outage that began at 14:10 UTC. Our investigation confirms that the incident caused increased request failures and delays in processing renders. Traffic levels were stable leading up to the incident, with no sudden increase in requests. We suspect Redis encountered an issue under sustained high load, but the underlying cause has not yet been confirmed. We have reduced load on the affected infrastructure as a precaution and are continuing to investigate with our provider. We are sorry for the disruption and will share further findings when available.