BitTitan incident

Impact to O365 migrations when using Impersonation permissions

Major Resolved View vendor source →

BitTitan experienced a major incident on May 21, 2021 affecting MigrationWiz, lasting 11d 13h. The incident has been resolved; the full update timeline is below.

Started
May 21, 2021, 05:03 PM UTC
Resolved
Jun 02, 2021, 06:33 AM UTC
Duration
11d 13h
Detected by Pingoru
May 21, 2021, 05:03 PM UTC

Affected components

MigrationWiz

Update timeline

  1. investigating May 21, 2021, 05:03 PM UTC

    We are aware that some customers are experiencing the following error when migrating mailboxes to an O365 destination. "An internal server error occurred. The operation failed., Object reference not set to an instance of an object." Microsoft is aware of this issue and is working towards a fix. We are working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible. We sincerely apologize for any inconvenience caused. Some affected customer admins may be able to see Microsoft's advisory in the MS Office Portal with incident number EX257082. This affects all of our MigrationWiz portals.

  2. investigating May 21, 2021, 07:02 PM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible

  3. investigating May 21, 2021, 09:01 PM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible

  4. investigating May 21, 2021, 11:18 PM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible

  5. investigating May 22, 2021, 02:34 AM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible

  6. investigating May 23, 2021, 02:35 AM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible

  7. investigating May 24, 2021, 02:18 AM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible

  8. investigating May 24, 2021, 01:58 PM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible. We appreciate this is impacting our customers and apologise for any inconvenience caused

  9. investigating May 24, 2021, 10:05 PM UTC

    We are still working urgently with Microsoft to seek guidance and an ETA on a permanent resolution and to ascertain any workaround available for customers. We will be providing further updates as soon as possible. We appreciate this is impacting our customers and apologize for any inconvenience caused

  10. identified May 24, 2021, 10:27 PM UTC

    The cause of the issue has been identified by Microsoft and we are working with them to develop a deployment plan for resolution as soon as possible and will provide further updates as information becomes available. We appreciate this is impacting our customers and apologize for any inconvenience caused

  11. identified May 25, 2021, 07:24 AM UTC

    The cause of the issue has been identified by Microsoft and we are working with them to develop a deployment plan for resolution as soon as possible and will provide further updates as information becomes available. We appreciate this is impacting our customers and apologize for any inconvenience caused

  12. identified May 25, 2021, 01:04 PM UTC

    The cause of the issue has been identified by Microsoft and we are working with them to develop a deployment plan for resolution as soon as possible and will provide further updates as information becomes available. Microsoft have released an advisory with further information (EX257082) and this can be downloaded from the following: https://bittitan.sharefile.com/d-s9150a59c5da74ef5a4a9a1fb8ce9b092 We will be providing further updates as soon as possible

  13. identified May 25, 2021, 04:20 PM UTC

    We have had confirmation from Microsoft that a fix is in the process of being deployed to affected tenants. Note that it is expected to take up to 8 days for this to be 100% deployed to all tenants. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will provide any further information as soon as we receive it from Microsoft Microsoft released an advisory earlier with further information (EX257082) and this can be downloaded from the following: https://bittitan.sharefile.com/d-s9150a59c5da74ef5a4a9a1fb8ce9b092 We will continue to provide further updates as soon as possible.

  14. identified May 25, 2021, 10:54 PM UTC

    The fix from Microsoft for this issue is still being deployed to affected tenants at this time. Note that it is expected to take up to 8 days for this to be 100% deployed to all tenants. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will provide any further information as soon as we receive it from Microsoft Microsoft released an advisory earlier with further information (EX257082) and this can be downloaded from the following: https://bittitan.sharefile.com/d-s9150a59c5da74ef5a4a9a1fb8ce9b092 We will continue to provide further updates as soon as possible.

  15. identified May 26, 2021, 10:25 AM UTC

    The fix from Microsoft for this issue is still being deployed to affected tenants at this time. Note that it is expected to take up to 7 days from now for this to be 100% deployed to all tenants. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will provide any further information as soon as we receive it from Microsoft Microsoft released an advisory earlier with further information (EX257082) and this can be downloaded from the following: https://bittitan.sharefile.com/d-s9150a59c5da74ef5a4a9a1fb8ce9b092 We will continue to provide further updates as soon as possible.

  16. identified May 26, 2021, 04:29 PM UTC

    We are continuing to work on a fix for this issue.

  17. identified May 26, 2021, 04:31 PM UTC

    Microsoft have provided an update that the fix for this issue is in the process of being deployed to affected tenants at this time. Note that it is expected to take up to 7 days from now for this to be 100% deployed to all tenants. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  18. identified May 27, 2021, 02:50 AM UTC

    Microsoft have provided an update that the fix for this issue is in the process of being deployed to affected tenants at this time. Note that it is expected to take up to 7 days from now for this to be 100% deployed to all tenants. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  19. identified May 27, 2021, 11:23 AM UTC

    Microsoft have provided an update that the fix for this issue is in the process of being deployed to affected tenants at this time. Note that it is expected to take up to 6 days from now for this to be 100% deployed to all tenants. Although we have had some reports where this has been deployed to tenants earlier and resolved the issue. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  20. identified May 27, 2021, 06:35 PM UTC

    The cause of the issue has been identified by Microsoft and we are working with them to develop a deployment plan for resolution as soon as possible and will provide further updates as information becomes available. We appreciate this is impacting our customers and apologize for any inconvenience caused

  21. identified May 28, 2021, 03:54 AM UTC

    Microsoft have provided an update that the fix for this issue is in the process of being deployed to affected tenants at this time. Note that it is expected to take up to 6 days from now for this to be 100% deployed to all tenants. Although we have had some reports where this has been deployed to tenants earlier and resolved the issue. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  22. identified May 28, 2021, 02:20 PM UTC

    Microsoft are continuing to deploy the fix for this issue to affected tenants. This could take a further 5 days from now for this to be 100% deployed to all tenants. Although we have had some reports where this has been deployed to tenants earlier and has resolved the issue. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  23. identified May 29, 2021, 08:08 AM UTC

    Microsoft are continuing to deploy the fix for this issue to affected tenants. This could take a further 4 days from now for this to be 100% deployed to all tenants. We have had some reports where this has been deployed to tenants already and has resolved the issue. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  24. identified May 30, 2021, 08:03 AM UTC

    Microsoft are continuing to deploy the fix for this issue to affected tenants. This could take a further 3 days from now for this to be 100% deployed to all tenants. We have had some reports where this has been deployed to tenants already and has resolved the issue. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. In the meantime we will continue to provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  25. monitoring Jun 01, 2021, 07:12 AM UTC

    Microsoft are continuing to deploy the fix for this issue to affected tenants. This could take 1 more day from now for this to be 100% deployed to all tenants. We have had reports where this has been deployed to tenants already and has resolved the issue. If you have been affected by this problem please continue to retry the migration to verify that the fix has been deployed. We will provide any further information as soon as we receive it from Microsoft. Thank you for your understanding.

  26. resolved Jun 02, 2021, 06:33 AM UTC

    All indications show that the deployment of the fix has been completed by Microsoft and that this issue has been resolved for our customers.