Arkose Labs incident
Arkose Global Bot Manager Errors And Latency
Arkose Labs experienced a critical incident on December 30, 2024 affecting North Virginia Healthcheck, lasting 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 30, 2024, 04:21 PM UTC
We are currently investigating an issue where some customers are seeing increased errors and latency in some regional endpoints.
- investigating Dec 30, 2024, 05:21 PM UTC
We are currently addressing platform issues, with some impact observed in regions including US East 1. While the error rate has decreased significantly, a small number of 503 errors are still being reported. We understand the importance of resolving this quickly and have our team fully engaged, with all hands on deck to ensure the platform’s stability. We have identified some issues within our platform we believe to be contributing to the impact. We are working at stabilizing those services to reduce their overall impact and will continue to troubleshoot after that. Our priority is to restore full functionality as quickly and efficiently as possible. Once the immediate resolution is complete, we will take the time to conduct a detailed post-mortem to analyze the incident thoroughly.
- identified Dec 30, 2024, 05:46 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Dec 30, 2024, 05:53 PM UTC
We have identified that the cause of this incident was resource contention among shared services. We isolated the service causing contention which freed up resources to allow the platform to recover and enter a healthy state again. We are continuing to monitor and conduct a full root cause analysis on this Incident. The Incident impact is now mitigated.