Ranking issues

Update timeline

1. investigating May 05, 2020, 07:40 PM UTC

   We're currently investigating an issue with Campfire ranking and 429 errors. We're working closely with our DevOps Team to resolve the issues as quickly as possible.

2. investigating May 05, 2020, 07:54 PM UTC

   We are continuing to investigate this issue.

3. investigating May 05, 2020, 07:56 PM UTC

   We've temporarily paused this service as we are facing issues on our backend database. We work with our team to resolve the issue.

4. investigating May 05, 2020, 07:56 PM UTC

   We are continuing to investigate this issue.

5. investigating May 05, 2020, 08:01 PM UTC

   We have no ETA of when this can be resolved. The issues are beyond the scope of Campfire, and are with our providers.

6. identified May 05, 2020, 08:36 PM UTC

   We've identified the root cause as a denial of service attack on our ranking service. We are taking security precautions to resolve the issue.

7. identified May 06, 2020, 11:40 AM UTC

   Our team are working on implementing rate limiting technology. We won’t ever hard rate limit, we’ll simply slow down your requests in transit. This is to prevent us from sending too many requests to Roblox’s servers. We will publish documentation to our Developer Docs when this has been implemented. Service will resume once this has been implemented. Thank you for your patience and cooperation.

8. monitoring May 06, 2020, 01:08 PM UTC

   A fix has been deployed. We are now monitoring the fix.

9. resolved May 06, 2020, 03:43 PM UTC

   This incident has been resolved.