EWS backup update

Update timeline

1. identified Apr 12, 2024, 05:48 AM UTC

   Keepit uses the Microsoft EWS protocol to read some data types. Microsoft has recently made a change in their back-end systems that has increased the volume of data EWS is returning for certain mailbox queries by a factor of 10-100x. We are working closely with Microsoft to help them understand the impact of this change, and hopefully revert it. Other data protection vendors are also affected. Temporarily, we have lowered the priority assigned to backup jobs of archive mailboxes to ensure that primary backups continue to run in a timely way. We will provide ongoing updates here as our investigation and remediation continue.

2. identified Apr 16, 2024, 02:22 PM UTC

   We are seeing an even increased inflow of EWS data and are continuing to work with Microsoft to understand and solve the issue

3. identified Apr 17, 2024, 05:46 AM UTC

   The situation is now improving across many regions. Archive mailboxes are still backed up at a lower priority.

4. identified Apr 17, 2024, 11:34 AM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Apr 18, 2024, 06:40 PM UTC

   Archive mailboxes are now being backed up again and data transport rates have normalized.

6. monitoring Apr 19, 2024, 05:45 PM UTC

   EWS backups have improved and we are monitoring the situation across all regions.

7. monitoring Apr 21, 2024, 07:22 AM UTC

   Backups execute timely and we are monitoring the situation across all regions.

8. monitoring Apr 23, 2024, 11:08 AM UTC

   Backups execute timely and we are monitoring the situation across all regions.

9. monitoring Apr 25, 2024, 11:59 AM UTC

   Backups have been executing timely for multiple days; we are monitoring the situation across all regions.

10. resolved May 06, 2024, 10:23 AM UTC

    The EWS situation has stabilised and is no longer impacting backup cadence.