Cronofy experienced a minor incident on September 21, 2023 affecting Google and Background Processing, lasting 1h 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 21, 2023, 01:40 PM UTC
We are seeing a significantly higher than normal amount of errors when trying to interact with Google calendars.
- monitoring Sep 21, 2023, 01:46 PM UTC
Errors when communicating with Google calendar increased significantly across all data centers from 13:31 UTC. We have taken action to reduce the impact on other calendar providers and continue to monitor.
- monitoring Sep 21, 2023, 02:51 PM UTC
We continue to observe a high error rate on Google’s Calendar API. This appears to affect some user cohorts more than others, rather than being spread evenly across all of our syncronized Google calendars. While this doesn’t affect the error rate of Cronofy’s API, it means that we’re failing to pick up the latest changes to affected users’ calendars, and may present stale availability information. Events written via the Cronofy API may be delayed before being pushed successfully to the external Google calendar. We queue and retry such operations automatically and do not anticipate any remedial action to be necessary as a result. We'll continue to monitor things and update accordingly.
- monitoring Sep 21, 2023, 03:55 PM UTC
Google’s Calendar API is still returning a high error rate, with no notable change to the situation since the previous update. Google are tracking the incident at their incident dashboard at https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ We’ll continue to monitor the situation and update here.
- monitoring Sep 21, 2023, 04:17 PM UTC
We're observing a much improved success rate from Google's Calendar API since 16:00 UTC. Google have advised that a fix has been made on their side and is rolling out currently, and expect to be fully resolved within the hour: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ We’ll continue to monitor the situation and update here.
- monitoring Sep 21, 2023, 04:45 PM UTC
Error rates from Google's API have returned to normal levels, and calendar syncs for Google-backed calendars are healthy again. Google Calendar sync performance dropped starting at 13:30 UTC, and was heavily degraded until improvements were seen starting at 16:00 UTC. A full recovery of service was reached around 16:10 UTC.
- resolved Sep 21, 2023, 05:31 PM UTC
Calendar sync for Google-backed calendars has remained healthy since the previous message, so we are considering this as resolved. Google have updated their incident record of the underlying issue, where they likewise consider it resolved: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ