Arkose Labs experienced a minor incident on April 11, 2024 affecting North Virginia Healthcheck and Ireland Healthcheck, lasting 12h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 11, 2024, 04:31 PM UTC
We have detected a potential degradation in US-EAST-1 and EU-WEST-1 region. Users may experience increase in 5xx errors. We are actively investigating the issue and will provide an update shortly.
- investigating Apr 11, 2024, 05:41 PM UTC
We are continuing to investigate this issue
- investigating Apr 11, 2024, 06:42 PM UTC
Initial investigation indicates that the increased HTTP 500 errors is stemming from a database migration that was completed at approximately 4:45 AM UTC on Thursday, April 11th. Our metrics show that the error rate is approximately 0.1% which is elevated compared to prior to the migration. We are continuing to investigate.
- investigating Apr 11, 2024, 07:49 PM UTC
We are continuing to investigate this issue.
- investigating Apr 11, 2024, 10:02 PM UTC
We are continuing to investigate this issue.
- investigating Apr 12, 2024, 12:23 AM UTC
We have narrowed down the issue and working towards a fix within our minor environments and will provide an update in 3 hours.
- monitoring Apr 12, 2024, 03:41 AM UTC
We have released a fix to address API_REQUEST_ERRORs (4xx’s errors), and we have observed improvements, returning to normal and continue monitoring.
- resolved Apr 12, 2024, 04:35 AM UTC
This incident has been resolved.