Cronofy experienced a minor incident on January 25, 2023 affecting Background Processing and Microsoft 365, lasting 2h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jan 25, 2023, 07:35 AM UTC
We are seeing a higher than usual number of errors affecting calendars hosted on Microsoft 365 across all our data centers for connections using Exchange Web Services (EWS) and Microsoft's Graph API. Background processing has scaled up to compensate but may be degraded as a result.
- monitoring Jan 25, 2023, 08:00 AM UTC
Microsoft have sent a 365 health notification (MO502273) relating to some users being unable to access multiple Microsoft 365 services. This is likely related to the issue we are seeing. Errors for 365-backed calendar continue to be elevated.
- monitoring Jan 25, 2023, 08:44 AM UTC
Error rates for Microsoft 365-backed calendars have returned to normal levels, and we have processed a surge in calendar updates. We will continue to monitor the service but believe this issue to now be resolved.
- resolved Jan 25, 2023, 10:18 AM UTC
The performance of syncing Microsoft 365-backed calendars was degraded for about 90 minutes. Microsoft 365 began returning an elevated number of errors and timing out just after 07:05 UTC until 08:30 UTC. This was part of a wider issue within Azure, where Microsoft 365 is hosted. Steps were taken to increase capacity and be more aggressive in timing out connections to Microsoft 365 to mitigate the impact on background processing. The sudden change to the volume of background jobs did lead to a slight delay in processing being present for much of the issue. Normal service resumed a few minutes after the Microsoft 365 APIs began to operate normally, once a surge of calendar update notifications was processed. We will be reviewing our mechanisms for reducing the side effects of such incidents in the future.