Is Cronofy down?

We are investigating an issue where Event Trigger notifications are failing to send in our US data center

A failing component in our US data center led to scheduled notifications for Event Triggers backing up and not sending. We've resolved the immediate cause, which will lead to notifications being delivered behind schedule. We are monitoring this and investigating any wider impact from the issue.

All pending notifications have now been flushed. Notifications scheduled since approximately 02:00 UTC on July 1st were failing to be sent. This affected customers in the US data center making use of the following API features: - Event Triggers; approximately 9000 notifications for calendar events starting/ending will have been delivered late - Scheduler Workflows; a small number of Scheduling Request lifecycle notifications will have been delivered late - Meeting Agents; a small number of scheduled Meeting Agents will not have arrived to their scheduled meetings A postmortem of the incident will take place and be attached to this incident in the next 48 hours.

Between approximately 02:25 UTC on July 1st and 10:11 UTC on July 7th 2026, scheduled notifications in our US data center failed to send. The affected notifications were queued but never picked up, and were flushed once the underlying cause was cleared. The impact was limited to customers in the US data center using notification-driven API features. Approximately 9,000 Event Trigger notifications for calendar events starting and ending were delivered late. A small number of Scheduler Workflows lifecycle notifications were also delivered late, and a small number of scheduled Meeting Agents did not arrive to their meetings. The root cause was a regression introduced on April 22nd, in an internal service responsible for sending scheduled notifications. The service uses leader election so that a single process handles distributing the imminent notifications at any one time, coordinated through a lock. The regression made that configuration fragile. If a leader did not release the lock cleanly, no other process would override it. When the leader stopped in the early hours of July 1st, it left the lock held, and with no process able to take over, notifications quietly backed up from that point. Nothing alerted us to this. The failure mode was an absence of activity rather than an error, and we had no alerting on notification throughput that would have caught the drop-off. During the incident, we manually cleared the stuck lock, which allowed the service to resume and flush the backlog within four minutes. ## Timeline All times are in UTC. * **April 22nd, 10:27** A change is merged that makes the leader election's lock-holding fragile. The fault is latent and has no immediate effect. * **July 1st, 02:25** The leader process in the US stops in a way that invokes the regression, leaving the lock held. No notifications are picked up from this point. * **July 2nd, 06:42** A support ticket reports an issue with Event Triggers. * **July 6th, 14:53** A second support ticket reports an issue with Event Triggers. * **July 7th, 09:38** The pattern across the two tickets is identified internally and raised. * **July 7th, 09:50** Telemetry is reviewed, showing a clear drop-off in notification sends. * **July 7th, 09:57** An internal incident channel is opened. * **July 7th, 10:05** The held lock is raised as the probable cause. * **July 7th, 10:07** The lock is manually cleared. * **July 7th, 10:07-10:11** The backlog of late notifications is flushed by the now-working service. * **July 7th, 10:12** The public incident is opened. * **July 7th, 10:40** Impact is reviewed and confirmed, including the other affected paths \(Scheduler Workflows, Meeting Agents\). * **July 7th, 11:12** The public incident is resolved. ## Retrospective We always ask the same three questions. ### Could the issue have been resolved sooner? Once we were looking at the right thing, resolution was fast. The probable cause was raised within eight minutes of opening the channel, and the backlog was flushed four minutes after that. We are satisfied with how the resolution itself went. ### Could the issue have been identified sooner? This is where we fell short, and badly. The failure was live for almost six days. Two things let it run that long. First, we had no alerting on notification throughput, so a complete stop in sending produced no signal. It is harder to alert on things _not_ happening, but that is exactly the case this warranted. Second, the first support ticket arrived on July 2nd and did not surface the pattern. It took a second ticket, four days later, for the connection to be made. Six days is too long for a customer-facing feature to be silently failing. ### Could the issue have been prevented? The regression was introduced in April and sat latent until a leader failed to shut down cleanly for the first time. The fragility was in the lock's release path, the case where a leader stops without releasing cleanly, which is precisely the kind of edge that is easy to miss in review and under-covered by tests. We did not catch it at either stage. ## Actions We have added alerting on notification throughput, so that a drop-off in sends is positively and quickly identifiable rather than depending on customer reports. We have fixed the code so that a lock held by a stopped leader is reliably released and picked up by another process, rather than leaving the work stuck. We have added tests around the configuration to ensure we cannot fall foul of the misconfiguration by accident, and integration tests to show that an orphaned lock can be obtained by another process. We are reviewing how we handle support tickets that describe an absence of expected behavior, so that a single ticket of this kind prompts a check of the relevant telemetry sooner.

We've been monitoring an issue on a cache cluster node since shortly after the maintenance at the weekend. As part of mitigation, customers may see increased errors from the API for a short time period. We will begin the mitigation momentarily.

The mitigation has been completed. Increased API errors were observed from 09:30:30 to 09:31:30 UTC.

We have seen that Google Meet Meeting Agents are being rejected from the meetings before being able to join and are investigating the issue. These meetings will fail to be recorded and transcribed.

We are rolling out a fix which has tested positively internally, and will be monitoring success rates

We are continuing to monitor for any further issues.

Google Meet Meeting Agents are making join attempts successfully since the fix release earlier. We will continue to monitor their performance.

After monitoring we've seen that the fix has improved performance but we're still seeing elevated cases of failures to join Google Meet Meetings, so are still working improving things further.

We are now rolling out a further fix which has tested well internally, and will continue to monitor its performance.

Google Meet Meeting Agents are attempting meeting joins successfully again after our further fix earlier today. We shall continue to monitor but consider this incident to be closed.

Google's API is returning an increased number of 503 errors when we attempt to sync calendars. This affects a small percentage of calendars, and some users may see delayed calendar syncs as a result. We're continuing to monitor.

We are continuing to see elevated errors when communicating with Google Calendar. Cronofy is operating normally - event changes are still syncing correctly. The only impact is that some changes may take slightly longer to be reflected in Google Calendars than usual. We are continuing to monitor the situation, but will only update this incident if there is a significant change in either direction.

Events continue to sync events into Google Calendars as normal. We are still seeing an elevated number of errors between Cronofy and Google Calendar, but are confident that our system are keeping events in sync correctly. The vast majority of events are still syncing almost immediately, while a minority of updates sync after a few seconds. This should have no impact on customers applications, so we are resolving the incident. We will continue to monitor the situation and will raise a new incident if performance declines to a noticeable level.

Between 03:08 UTC and 03:10 UTC we observed increased API error rates in our US data center. During this time, customers may have received 500 Internal Server Error responses. Since that time, error rates have returned to normal levels. These errors were caused by a read-only database server going through an automated recovery process.