Immutable incident

Batch processing delays impacting withdrawal times

Minor Resolved View vendor source →

Immutable experienced a minor incident on August 23, 2024 affecting Staking Rewards, lasting 16d 23h. The incident has been resolved; the full update timeline is below.

Started
Aug 23, 2024, 12:08 AM UTC
Resolved
Sep 09, 2024, 12:00 AM UTC
Duration
16d 23h
Detected by Pingoru
Aug 23, 2024, 12:08 AM UTC

Affected components

Staking Rewards

Update timeline

  1. investigating Aug 23, 2024, 12:08 AM UTC

    Withdrawal and batch processing times are expected to take up to 48 hours to be in the 'Ready to Withdraw' state. We will continue to monitor this issue until normal processing times resume. A further update will be provided at 6:00AM UTC on the 23rd August 2024

  2. identified Aug 23, 2024, 06:37 AM UTC

    We are continuing to investigate remediation options for batch processing and withdrawal delays. The backlog is progressing and expected resolution time frames are still accurate.

  3. identified Aug 23, 2024, 10:17 PM UTC

    We continue to monitor the backlog of withdrawals to ensure all transactions are processed. Currently the batch processing times are up to 60 hours for withdrawals. A further update will be provided at 10:00PM UTC on the 24th August 2024 Thank you for your patience.

  4. identified Aug 25, 2024, 12:05 AM UTC

    We are continuing to investigate remediation options for batch processing and withdrawal delays. Withdrawals are progressing and the current expectation from Submitted to 'Ready to Withdraw' is 80 hours.

  5. identified Aug 25, 2024, 05:49 AM UTC

    The delay for withdrawal processing from Immutable X has has now reduced to approximately 72 hours. This backlog was caused by a high volume of transactions. We are hoping to reduce this to the standard <8 hour delay very soon. Further updates will be provided as this changes.

  6. identified Aug 26, 2024, 07:20 AM UTC

    Due to a continued high volume of transactions, withdrawals from Immutable X are currently expected to take 72 hours. Withdrawal via layerswap continues to be available for supported tokens. Our team is still working toward reducing withdrawal processing time to the standard <8 hour delay as soon as possible. Further updates will be provided as this changes.

  7. identified Aug 27, 2024, 08:37 AM UTC

    Our team is continuing to monitor the high volume of transactions currently on Immutable X. Withdrawals are currently expected to be completed processing within 72 hours. Withdrawal via layerswap continues to be available for supported tokens. Further updates will be provided as this changes.

  8. identified Aug 28, 2024, 10:52 AM UTC

    Our team is continuing to monitor the high volume of transactions currently on Immutable X. Withdrawals from Immutable X continued to be delayed, with expected processing completion times at 72 hours. Withdrawal via layerswap continues to be available for supported tokens. Further updates will be provided as this changes. Thank you for your patience.

  9. identified Aug 29, 2024, 05:29 AM UTC

    We are continuing to work on a fix for this issue.

  10. identified Sep 02, 2024, 06:37 AM UTC

    Our team is continuing to monitor the high volume of transactions currently on Immutable X. Withdrawals are currently expected to be completed processing within 72 hours. Withdrawal via layerswap continues to be available for supported tokens. Further updates will be provided as this changes.

  11. identified Sep 05, 2024, 12:08 AM UTC

    Due to ongoing batch processing delays, withdrawals from Immutable X are currently expected to take a minimum of 72 hours, and may take up to 96 hours. We expect to return to the normal 8 hour processing time of withdrawals by 8th September 2024. Withdrawal via layerswap continues to be available for supported tokens. Further updates will be provided as this changes.

  12. monitoring Sep 06, 2024, 04:52 AM UTC

    We are happy to report the backlog of pending withdrawals has now been processed. We expect these to be available for users in 'Ready to Withdraw' status from Immutable X within the next 24 hours (maximum). We would like to pass on our apologies for the delays users have experienced during this time. An exceptionally high volume of minting and transfer activity resulted in more than 15 million transactions in a 48 hour window. This volume delayed all batch processing and flowed on to impact withdrawals from Immutable X. Our team is working through a deeper retrospective to look for ways we can expedite withdrawals in future to avoid this impact.

  13. resolved Sep 09, 2024, 12:00 AM UTC

    This incident has now been resolved.