Arkose Labs incident

Potential Degradation of Setup Session

Resolved after 12 hours 3 minutes, affecting North Virginia Healthcheck and Ireland Healthcheck, on April 11, 2024.

monitoring Apr 12, 2024, 03:41 AM UTC

We have released a fix to address API_REQUEST_ERRORs (4xx’s errors), and we have observed improvements, returning to normal and continue monitoring.

Started
Apr 11, 2024, 04:31 PM UTC
Resolved
Apr 12, 2024, 04:35 AM UTC
Duration
12h 3m
Detected by Pingoru
Apr 11, 2024, 04:31 PM UTC

Affected components

North Virginia HealthcheckIreland Healthcheck

Update timeline

  1. investigating Apr 11, 2024, 04:31 PM UTC

    We have detected a potential degradation in US-EAST-1 and EU-WEST-1 region. Users may experience increase in 5xx errors. We are actively investigating the issue and will provide an update shortly.

  2. investigating Apr 11, 2024, 05:41 PM UTC

    We are continuing to investigate this issue

  3. investigating Apr 11, 2024, 06:42 PM UTC

    Initial investigation indicates that the increased HTTP 500 errors is stemming from a database migration that was completed at approximately 4:45 AM UTC on Thursday, April 11th. Our metrics show that the error rate is approximately 0.1% which is elevated compared to prior to the migration. We are continuing to investigate.

  4. investigating Apr 11, 2024, 07:49 PM UTC

    We are continuing to investigate this issue.

  5. investigating Apr 11, 2024, 10:02 PM UTC

    We are continuing to investigate this issue.

  6. investigating Apr 12, 2024, 12:23 AM UTC

    We have narrowed down the issue and working towards a fix within our minor environments and will provide an update in 3 hours.

  7. monitoring Apr 12, 2024, 03:41 AM UTC

    We have released a fix to address API_REQUEST_ERRORs (4xx’s errors), and we have observed improvements, returning to normal and continue monitoring.

  8. resolved Apr 12, 2024, 04:35 AM UTC

    This incident has been resolved.

Your product runs on more than Arkose Labs

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →