Cronofy Outage History

The mitigation has been completed. Increased API errors were observed from 09:30:30 to 09:31:30 UTC.

Google Meet Meeting Agents are attempting meeting joins successfully again after our further fix earlier today. We shall continue to monitor but consider this incident to be closed.

Events continue to sync events into Google Calendars as normal. We are still seeing an elevated number of errors between Cronofy and Google Calendar, but are confident that our system are keeping events in sync correctly. The vast majority of events are still syncing almost immediately, while a minority of updates sync after a few seconds. This should have no impact on customers applications, so we are resolving the incident. We will continue to monitor the situation and will raise a new incident if performance declines to a noticeable level.

We have released version 0.22.25 of the Cronofy Chrome extension, fixing the issue. If you have any problems, please first check that both Google Chrome and the Cronofy Chrome extension are up to date. If you have any additional issues then please contact [email protected].

Between 12:52 and 12:55 we saw that calls to our API encountered a higher level of 500 Server Error responses than usual We have investigated this and it appears to relate to a period where we saw a large spike in traffic, a rapid drop and another spike in quick succession resulting in some temporary 500 errors. This is now resolved but we will continue monitoring for any further issues.

We have seen that our error rate provisioning MS Teams Conferencing has returned to normal, low levels since 16:10 UTC yesterday. Normal levels of service have resumed, and we're working to improve our monitoring so that similar issues are detected sooner in the future.

Between 22:00 and 01:15 UTC we saw elevated error rates when performing sync operations with Microsoft 365. Microsoft continues to work on a full resolution to the incident with ID MO1221364. However, the error rates we are now observing are back to normal levels.

Between 18:15 UTC and 18:28 UTC, we observed elevated error rates, and Google Calendar users may have experienced delays with calendar reads and writes. Since 18:28 UTC, all Google Calendar operations have been functioning as normal.

Starting at 10:40 UTC we observed an elevated error rate when communicating with Apple's calendar service. Errors returned to regular levels by 11:35 UTC. Sync with Apple calendars may have taken longer both to and from Apple as a result of the underlying errors during this period.

API success rates have remained stable since our previous message, and Google have resolved their own incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9 Between the times of 14:57 UTC - 15:35 UTC, we saw elevated rates of errors, and Google Calendar users may have had slight delays to calendar reads and writes. Since 15:35 UTC everything has been functioning as normal.

Microsoft Issue ID EX1112414 remains open however a preliminary root cause has been provided: "A recent service update to an authentication component is unintentionally preventing access for a subset of users, resulting in intermittent service unavailability." We've seen no further issues since our previous update and error rates remain consistent with healthy operations so we're going to consider this incident resolved.

Error rates have remained at normal levels and we believe the issue is over. After two increases in error rates, at 14:24 UTC and 14:45 UTC, Google sync has remained stable. A minority of sync operations failed and were retried, eventually completing successfully.

Google API calls began to error at abnormally high rates from 17:49 UTC. Upon investigation, we did not find a common factor or pattern behind the failures but kept investigating and engages some fallback mechanisms to keep the platform stable. We began to find other reports of Google API issues, ultimately having this confirmed by Google: - Google Workspace incident: https://www.google.com/appsstatus/dashboard/incidents/Eab7zGLWSaUJyh9c9R9L - Google Cloud incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4 We continued to monitor the system and confirmed that calendars were still syncing, albeit with occasional delays when syncs failed and had to be retried. At 18:48 UTC, API errors dropped drastically and returned to normal operational levels. We kept monitoring until we saw Google report the issue was resolved at 19:53 UTC.