Ikeono incident

Intermittent time-outs on Ikeono Portal

Minor Resolved View vendor source →

Ikeono experienced a minor incident on August 14, 2023 affecting Portal, lasting 6h 5m. The incident has been resolved; the full update timeline is below.

Started
Aug 14, 2023, 04:58 PM UTC
Resolved
Aug 14, 2023, 11:04 PM UTC
Duration
6h 5m
Detected by Pingoru
Aug 14, 2023, 04:58 PM UTC

Affected components

Portal

Update timeline

  1. identified Aug 14, 2023, 04:58 PM UTC

    A downstream provider is currently having issues leading to intermittent time-outs on the Ikeono portal on the East Coast. They are actively working on a fix, and we will update as we have more information.

  2. identified Aug 14, 2023, 06:52 PM UTC

    We are continuing to press our downstream provider for a fix. The errors are now happening across the US, not just on the East Coast.

  3. identified Aug 14, 2023, 07:36 PM UTC

    Regional Issues from this morning has created a large backlog that is being worked through currently. We are pushing updates to deprioritize features that are not business critical, to help the main Ikeono portal load faster.

  4. identified Aug 14, 2023, 09:02 PM UTC

    We are continuing to work on a fix for this issue.

  5. identified Aug 14, 2023, 09:02 PM UTC

    We are doing an emergency database migration. Expected down time is less than 10 minutes.

  6. identified Aug 14, 2023, 09:04 PM UTC

    We are doing an emergency database migration. Expected down time is less than 10 minutes.

  7. identified Aug 14, 2023, 09:13 PM UTC

    The migration is in process.

  8. monitoring Aug 14, 2023, 09:25 PM UTC

    The migration is complete

  9. resolved Aug 14, 2023, 11:04 PM UTC

    Everything has been resolved