Cronofy Outage History

Started Jul 07, 2026, 10:12 AM UTC· resolved Jul 07, 2026, 11:12 AM UTC

1. investigating Jul 07, 2026, 10:12 AM UTC

   We are investigating an issue where Event Trigger notifications are failing to send in our US data center

2. monitoring Jul 07, 2026, 10:20 AM UTC

   A failing component in our US data center led to scheduled notifications for Event Triggers backing up and not sending. We've resolved the immediate cause, which will lead to notifications being delivered behind schedule. We are monitoring this and investigating any wider impact from the issue.

3. resolved Jul 07, 2026, 11:12 AM UTC

   All pending notifications have now been flushed. Notifications scheduled since approximately 02:00 UTC on July 1st were failing to be sent. This affected customers in the US data center making use of the following API features: - Event Triggers; approximately 9000 notifications for calendar events starting/ending will have been delivered late - Scheduler Workflows; a small number of Scheduling Request lifecycle notifications will have been delivered late - Meeting Agents; a small number of scheduled Meeting Agents will not have arrived to their scheduled meetings A postmortem of the incident will take place and be attached to this incident in the next 48 hours.

4. postmortem Jul 08, 2026, 07:46 PM UTC

   Between approximately 02:25 UTC on July 1st and 10:11 UTC on July 7th 2026, scheduled notifications in our US data center failed to send. The affected notifications were queued but never picked up, and were flushed once the underlying cause was cleared. The impact was limited to customers in the US data center using notification-driven API features. Approximately 9,000 Event Trigger notifications for calendar events starting and ending were delivered late. A small number of Scheduler Workflows lifecycle notifications were also delivered late, and a small number of scheduled Meeting Agents did not arrive to their meetings. The root cause was a regression introduced on April 22nd, in an internal service responsible for sending scheduled notifications. The service uses leader election so that a single process handles distributing the imminent notifications at any one time, coordinated through a lock. The regression made that configuration fragile. If a leader did not release the lock cleanly, no other process would override it. When the leader stopped in the early hours of July 1st, it left the lock held, and with no process able to take over, notifications quietly backed up from that point. Nothing alerted us to this. The failure mode was an absence of activity rather than an error, and we had no alerting on notification throughput that would have caught the drop-off. During the incident, we manually cleared the stuck lock, which allowed the service to resume and flush the backlog within four minutes. ## Timeline All times are in UTC. * **April 22nd, 10:27** A change is merged that makes the leader election's lock-holding fragile. The fault is latent and has no immediate effect. * **July 1st, 02:25** The leader process in the US stops in a way that invokes the regression, leaving the lock held. No notifications are picked up from this point. * **July 2nd, 06:42** A support ticket reports an issue with Event Triggers. * **July 6th, 14:53** A second support ticket reports an issue with Event Triggers. * **July 7th, 09:38** The pattern across the two tickets is identified internally and raised. * **July 7th, 09:50** Telemetry is reviewed, showing a clear drop-off in notification sends. * **July 7th, 09:57** An internal incident channel is opened. * **July 7th, 10:05** The held lock is raised as the probable cause. * **July 7th, 10:07** The lock is manually cleared. * **July 7th, 10:07-10:11** The backlog of late notifications is flushed by the now-working service. * **July 7th, 10:12** The public incident is opened. * **July 7th, 10:40** Impact is reviewed and confirmed, including the other affected paths \(Scheduler Workflows, Meeting Agents\). * **July 7th, 11:12** The public incident is resolved. ## Retrospective We always ask the same three questions. ### Could the issue have been resolved sooner? Once we were looking at the right thing, resolution was fast. The probable cause was raised within eight minutes of opening the channel, and the backlog was flushed four minutes after that. We are satisfied with how the resolution itself went. ### Could the issue have been identified sooner? This is where we fell short, and badly. The failure was live for almost six days. Two things let it run that long. First, we had no alerting on notification throughput, so a complete stop in sending produced no signal. It is harder to alert on things _not_ happening, but that is exactly the case this warranted. Second, the first support ticket arrived on July 2nd and did not surface the pattern. It took a second ticket, four days later, for the connection to be made. Six days is too long for a customer-facing feature to be silently failing. ### Could the issue have been prevented? The regression was introduced in April and sat latent until a leader failed to shut down cleanly for the first time. The fragility was in the lock's release path, the case where a leader stops without releasing cleanly, which is precisely the kind of edge that is easy to miss in review and under-covered by tests. We did not catch it at either stage. ## Actions We have added alerting on notification throughput, so that a drop-off in sends is positively and quickly identifiable rather than depending on customer reports. We have fixed the code so that a lock held by a stopped leader is reliably released and picked up by another process, rather than leaving the work stuck. We have added tests around the configuration to ensure we cannot fall foul of the misconfiguration by accident, and integration tests to show that an orphaned lock can be obtained by another process. We are reviewing how we handle support tickets that describe an absence of expected behavior, so that a single ticket of this kind prompts a check of the relevant telemetry sooner.

Read the full incident report →

Started Jun 16, 2026, 09:25 AM UTC· resolved Jun 16, 2026, 09:56 AM UTC

1. identified Jun 16, 2026, 09:25 AM UTC

   We've been monitoring an issue on a cache cluster node since shortly after the maintenance at the weekend. As part of mitigation, customers may see increased errors from the API for a short time period. We will begin the mitigation momentarily.

2. resolved Jun 16, 2026, 09:56 AM UTC

   The mitigation has been completed. Increased API errors were observed from 09:30:30 to 09:31:30 UTC.

Read the full incident report →

Started Jun 03, 2026, 03:12 PM UTC· resolved Jun 04, 2026, 01:05 PM UTC

1. investigating Jun 03, 2026, 03:12 PM UTC

   We have seen that Google Meet Meeting Agents are being rejected from the meetings before being able to join and are investigating the issue. These meetings will fail to be recorded and transcribed.

2. monitoring Jun 03, 2026, 04:31 PM UTC

   We are rolling out a fix which has tested positively internally, and will be monitoring success rates

3. monitoring Jun 03, 2026, 06:30 PM UTC

   We are continuing to monitor for any further issues.

4. monitoring Jun 03, 2026, 06:34 PM UTC

   Google Meet Meeting Agents are making join attempts successfully since the fix release earlier. We will continue to monitor their performance.

5. monitoring Jun 04, 2026, 09:39 AM UTC

   After monitoring we've seen that the fix has improved performance but we're still seeing elevated cases of failures to join Google Meet Meetings, so are still working improving things further.

6. monitoring Jun 04, 2026, 11:00 AM UTC

   We are now rolling out a further fix which has tested well internally, and will continue to monitor its performance.

7. resolved Jun 04, 2026, 01:05 PM UTC

   Google Meet Meeting Agents are attempting meeting joins successfully again after our further fix earlier today. We shall continue to monitor but consider this incident to be closed.

Read the full incident report →

Started Jun 02, 2026, 01:40 PM UTC· resolved Jun 02, 2026, 08:29 PM UTC

1. monitoring Jun 02, 2026, 01:40 PM UTC

   Google's API is returning an increased number of 503 errors when we attempt to sync calendars. This affects a small percentage of calendars, and some users may see delayed calendar syncs as a result. We're continuing to monitor.

2. monitoring Jun 02, 2026, 04:55 PM UTC

   We are continuing to see elevated errors when communicating with Google Calendar. Cronofy is operating normally - event changes are still syncing correctly. The only impact is that some changes may take slightly longer to be reflected in Google Calendars than usual. We are continuing to monitor the situation, but will only update this incident if there is a significant change in either direction.

3. resolved Jun 02, 2026, 08:29 PM UTC

   Events continue to sync events into Google Calendars as normal. We are still seeing an elevated number of errors between Cronofy and Google Calendar, but are confident that our system are keeping events in sync correctly. The vast majority of events are still syncing almost immediately, while a minority of updates sync after a few seconds. This should have no impact on customers applications, so we are resolving the incident. We will continue to monitor the situation and will raise a new incident if performance declines to a noticeable level.

Read the full incident report →

Started Feb 27, 2026, 04:06 PM UTC· resolved Feb 27, 2026, 05:35 PM UTC

1. identified Feb 27, 2026, 04:06 PM UTC

   Since approximately 12:30 UTC a subset of calendar events failed to be synced to users' calendars. We've identified the source of the issue and released a fix, we are now assessing the impact and actions necessary to resolve the issue for previously impacted calendar events.

2. identified Feb 27, 2026, 04:28 PM UTC

   We have identified that affected events were those being written to Microsoft calendars and using the `integrated` conferencing option. The system is already retrying these failures and successfully writing them. We are looking to accelerate this process.

3. identified Feb 27, 2026, 05:00 PM UTC

   At approximately 12:30 UTC a change was released that prevented events being created in Microsoft calendars if integrated conferencing was being used. A fix was released and the last error observed at approximately 16:05 UTC. Since that time over 90% of the impacted events have been successfully synced due to standard retry mechanisms with the platform. We are working to enqueue additional attempts for all affected calendars to speed up this process and to be certain nothing has been missed.

4. resolved Feb 27, 2026, 05:35 PM UTC

   At approximately 12:30 UTC a change was released that prevented events being created in Microsoft calendars if integrated conferencing was being used. A fix was released and the last error observed at approximately 16:05 UTC. Over 90% of the impacted events were successfully synced due to standard retry mechanisms with the platform by 17:00 UTC, we also ran a sweep to proactively retry all remaining events which completed by 17:30 UTC. This incident is now resolved and operations are back to normal. We will be conducting a postmortem and will publish the findings of that next week.

5. postmortem Mar 03, 2026, 03:12 PM UTC

   On Friday February 27th 2026 at 12:28 UTC, a change was released to our production environment that caused a bug preventing events using Microsoft Integrated conferencing from being synced to calendars. At 16:05 UTC, a change was released to fix the issue and full functionality was restored. By 17:35 UTC, all previously impacted events had been successfully retried and synced and the incident was closed. In total, we identified 2,254 events that had been affected by this bug, all of which have now been successfully synced. We always ask the questions: * Could the issue have been identified sooner? * Could the issue have been resolved sooner? * Could the issue have been prevented? In this instance, the issue could and should have been identified by us earlier. The change that introduced this bug was released at 12:28 UTC, but we only became aware of the issue when a customer reported issues to us at 15:27 UTC. We never want to rely on customers having to tell us about issues in our platform before we are aware, so this is an instance where gaps in our system monitoring meant the issue went undetected for too long. We deployed a fix for this issue at 16:05 UTC, which was 37 minutes after the bug report email was sent, and 13 minutes after we identified the extent of the issue and opened an incident at 15:52 UTC. Therefore, we don’t believe the bug could’ve been resolved any sooner, as we acted quickly to release a fix once we were fully aware of the problem. Along with a gap in our alerting, the bug should have been prevented through automated testing, which would’ve flagged the issue before the initial change was deployed at 12:28 UTC. In this instance, our testing failed to check the logic down the path that Microsoft Teams integrated conferencing followed, because a check had been mocked instead of being integration tested, leading to a blind spot. We have addressed this and are reviewing where we have used test mocks which could hide integration issues. Actions: * Further guidelines around coding principles to be provided to our Engineering team regarding our approach to automated testing * Improve our system monitoring to provide specific alerting around this particular area

Read the full incident report →

Started Feb 17, 2026, 11:22 AM UTC· resolved Feb 17, 2026, 02:10 PM UTC

1. investigating Feb 17, 2026, 11:22 AM UTC

   We are aware of issues affecting our Google Chrome Extension preventing users from creating scheduling requests in Gmail. We are currently investigating and will provide an update as soon as possible.

2. investigating Feb 17, 2026, 12:01 PM UTC

   We are still working to identify the exact root cause of the issue. We will provide further updates as soon as possible.

3. identified Feb 17, 2026, 12:39 PM UTC

   A recent Google Chrome update has changed a mechanism that our Gmail extension relies on. We are developing a fix that is compatible with this new mechanism. We will provide an update in a few hours or when we have an updated version of the extension ready.

4. identified Feb 17, 2026, 01:34 PM UTC

   We have submitted an updated version of the Chrome Extension to Google which contains a fix for this issue. The updated version will be available as soon as it has been checked and published by Google. We will provide an update once we confirmation of this from Google.

5. resolved Feb 17, 2026, 02:10 PM UTC

   We have released version 0.22.25 of the Cronofy Chrome extension, fixing the issue. If you have any problems, please first check that both Google Chrome and the Cronofy Chrome extension are up to date. If you have any additional issues then please contact [email protected].

Read the full incident report →

Started Feb 13, 2026, 01:44 PM UTC· resolved Feb 13, 2026, 01:44 PM UTC

1. resolved Feb 13, 2026, 01:44 PM UTC

   Between 12:52 and 12:55 we saw that calls to our API encountered a higher level of 500 Server Error responses than usual We have investigated this and it appears to relate to a period where we saw a large spike in traffic, a rapid drop and another spike in quick succession resulting in some temporary 500 errors. This is now resolved but we will continue monitoring for any further issues.

Read the full incident report →

Started Feb 11, 2026, 11:29 AM UTC· resolved Feb 12, 2026, 10:59 AM UTC

1. investigating Feb 11, 2026, 11:29 AM UTC

   We are investigating an elevated level of failures for provisioning MS Teams Conferencing.

2. investigating Feb 11, 2026, 12:07 PM UTC

   We are continuing to investigate - the issue is not impacting all customers but we are not yet seeing a pattern to the domains or events impacted.

3. investigating Feb 11, 2026, 12:54 PM UTC

   We are still investigating; we are capturing additional logs to help us narrow down the failing population. Some conferences are being provisioned after automatically retrying, but still not as many as we'd expect.

4. investigating Feb 11, 2026, 01:31 PM UTC

   We've found some evidence that other users of the Microsoft Graph API are experiencing the same issue; we're looking for anything that may let us mitigate the impact and we're reaching out to Microsoft for support.

5. investigating Feb 11, 2026, 02:59 PM UTC

   We believe the root cause is a lack of generated URLs for a meeting when we call Microsoft Graph APIs. We are waiting for a response from Microsoft on the issue. Unfortunately the timing of this is out of our hands. We are also working on alternative ways to provision Teams meetings, but this will not provide a short term fix. We are continuing to work on finding a way to resolve, mitigate or provide an more reliable path to Teams provisioning. We will update this incident over the coming days as we make meaningful progress.

6. resolved Feb 12, 2026, 10:59 AM UTC

   We have seen that our error rate provisioning MS Teams Conferencing has returned to normal, low levels since 16:10 UTC yesterday. Normal levels of service have resumed, and we're working to improve our monitoring so that similar issues are detected sooner in the future.

Read the full incident report →

Started Jan 22, 2026, 11:18 PM UTC· resolved Jan 23, 2026, 01:19 AM UTC

1. investigating Jan 22, 2026, 11:18 PM UTC

   We are currently investigating degraded sync performance for Microsoft 365 as we are experiencing a much higher error rate than normal.

2. monitoring Jan 22, 2026, 11:28 PM UTC

   We believe this is related to a Microsoft incident (ID: MO1221364). We are keeping a close eye on how this progresses

3. monitoring Jan 23, 2026, 12:01 AM UTC

   We are continuing to monitor the situation, but have not seen any improvement so far

4. resolved Jan 23, 2026, 01:19 AM UTC

   Between 22:00 and 01:15 UTC we saw elevated error rates when performing sync operations with Microsoft 365. Microsoft continues to work on a full resolution to the incident with ID MO1221364. However, the error rates we are now observing are back to normal levels.

Read the full incident report →

Started Nov 18, 2025, 05:37 PM UTC· resolved Nov 18, 2025, 06:33 PM UTC

1. monitoring Nov 18, 2025, 05:37 PM UTC

   Between 16:55 UTC and 17:03 UTC we observed increased API error rates in our US data center. Error rates returned to normal by 17:03 UTC. We are monitoring and investigating the root cause.

2. monitoring Nov 18, 2025, 06:18 PM UTC

   Error rates remain at normal levels, and we are continuing to monitor.

3. resolved Nov 18, 2025, 06:33 PM UTC

   Between 16:55 UTC and 17:03 UTC we observed increased API error rates in our US data center. During this time customers may have received 401 Unauthorized or 500 Internal Server Error responses. Since that time error rates have returned to normal levels. A postmortem of the incident will take place and be attached to this incident in the coming days.

4. postmortem Nov 19, 2025, 03:41 PM UTC

   On November 18th 16:47UTC we had several errors about our database cluster encountering a sync issue. This ultimately resulted in an automated recovery mechanism kicking in, resulting in about 90 seconds where most communication with the database was blocked. The errors indicate that the root cause was an issue with the storage layer that the database relies on. No data was lost during this procedure, but customers will have seen our APIs respond with HTTP 500 errors as the database was unreachable. We posted an incident page as soon as we had investigated and understood that there were no ongoing issues. We have set up further alerting on these specific errors going forward to provide more thorough monitoring coverage. We will also shortly be announcing a database upgrade, planned prior to this incident, which will include stability improvements for our database technology. * `16:47` - We get the first database log message relating to replication * `16:54` - Database replica lag begins to climb * `16:55` - We start to see more of the replication errors * `16:56` - Internal alerting highlights increased client and server errors * `16:57` - We get a message stating that the database replica has been disconnected and reconnected to automatically resolve the issue. * `16:58` - Database replica lag returns to normal * `17:14` - Internal incident channel opened for investigation * `17:37` - [Public Status Page incident opened](https://status.cronofy.com/incidents/nmhfk7kl9t9m) * `17:37` - Monitored the database to make sure error levels had returned to normal * `18:33` - Status Page incident closed ### Retrospective * Could it have bee identified sooner? * Yes * The database replication error is rare enough that it should’ve been worth being notified of. Had we been notified of this, we would’ve known there were issues. * Could it have been resolved sooner? * No * Even if we had known about this earlier, we couldn’t have done anything other than what ended up happening to resolve the issue. * Could it have been prevented? * No * With our current understanding, the issue appears to be related to a database issue that was out of our control. Once the issue started happening we wouldn’t have been able to take any action to resolve as the issue essentially self healed. ### Actions * Set up more alerting for specific database replication errors. * Conduct further internal reviews to solidify our understand on the above errors.

Read the full incident report →

Started Nov 12, 2025, 06:31 PM UTC· resolved Nov 12, 2025, 06:56 PM UTC

1. investigating Nov 12, 2025, 06:31 PM UTC

   We are currently investigating the cause of Google calendar sync degradation

2. monitoring Nov 12, 2025, 06:40 PM UTC

   Google's API error rate has dropped again and has been at normal levels for the past 10 minutes. We'll continue to monitor and confirm this holds out.

3. resolved Nov 12, 2025, 06:56 PM UTC

   Between 18:15 UTC and 18:28 UTC, we observed elevated error rates, and Google Calendar users may have experienced delays with calendar reads and writes. Since 18:28 UTC, all Google Calendar operations have been functioning as normal.

Read the full incident report →

Started Oct 23, 2025, 10:35 AM UTC· resolved Oct 23, 2025, 10:59 AM UTC

1. investigating Oct 23, 2025, 10:35 AM UTC

   Customers using our Canadian data center may have experienced interruption of service when accessing both our API and application. This was caused by a recent change and will have presented as HTTP 502 - Gateway Timeout statuses. We have reverted the change and are in the process of assessing the exact scope of the issue.

2. resolved Oct 23, 2025, 10:59 AM UTC

   Between 10:12 and 10:18 UTC a change to the way back-end resources are allocated led to a scenario whereby the services processing requests began being forcibly restarted. This was caused by the services rapidly exceeding predefined resource limits and being taken out of service. This resulted in a partial outage that affected a small number of requests. Our internal monitoring alerted us to the issue and the changes were reverted. We're continuing to monitor the situation but consider this incident resolved. Full service was restored at 10:18 UTC.

3. postmortem Oct 27, 2025, 02:15 PM UTC

   ### **Summary** On October 23rd, between 10:11 and 10:18 UTC, we experienced a partial outage in our Canadian data center. During this time customers may have received server errors or experienced increased response latency when using the scheduler or our API. Background processing was not impacted during this incident. At 10:09 UTC, we released a change that increased how much work each individual web server process can handle at a time. This inadvertently caused those processes to hit resource limits after a short period of time, which then resulted in those processes being restarted. This then kept repeating until we reverted the change. The first errors happened at 10:11 UTC, and our engineers were alerted to this at 10:14. At 10:17 we reverted the change, and by 10:19 UTC the server errors had stopped and response latency had returned to normal. ### **Postmortem** We always ask the questions: * Could it have been resolved sooner * No, reverting the change happened very quickly after we were alerted to the errors. The errors stopped as soon as our deployment processed had rolled back * Could it have been identified sooner * No, alerts came through within minutes of the initial change being deployed. Our external monitoring tools also check the availability of our API and scheduler every minute. * Could it have been prevented * Yes * This was not tested in our test environment before deploying. Replicating production traffic loads in difficult, so had we deployed this in our test environment we may not have encountered the exact same issue. However, there may have been some signs of increase resource utilization. * Rather than increasing how much work each individual web server process can handle, we could have increased the number of processes running. This is known as scaling horizontally, rather than scaling vertically. * There was also an oversight on how resource utilization would be affected by this change. From this we have now learnt more about the nuance involved with making these specific changes. ### **Actions** * Make it harder to make this same mistake again by adding comments to the codebase to ensure if a similar change is made in the future that we also appropriately increase resource limits * We felt that we could have been quicker to communicate about the issue on our status page. To help with this in the future we will: * Update our Incident Playbook to ensure guidelines on communication timelines are clear to all * Give more engineers access to post and update incident on our status page

Read the full incident report →

Started Oct 20, 2025, 09:52 AM UTC· resolved Oct 20, 2025, 09:58 PM UTC

1. monitoring Oct 20, 2025, 09:52 AM UTC

   Since around 07:15 UTC our US data center has been severely impacted by an ongoing incident in AWS us-east-1, where it is hosted: https://health.aws.amazon.com/health/status AWS have identified a potential root cause for the issue and are working on multiple parallel paths to accelerate recovery. At 09:20 UTC we started seeing signs of recovery. As of 09:30 UTC all high priority tasks were cleared which are those with the most visible user impact. We are unable to scale additional capacity to clear lower priority tasks at present, which are more background processes such as polling calendars (Apple) and housekeeping style tasks. This backlog is being processed, but not at a pace that means we are clearing it as quickly as we would like. All other data centers are fully operational.

2. monitoring Oct 20, 2025, 10:29 AM UTC

   Since around 07:15 UTC our US data center has been severely impacted by an ongoing incident in AWS us-east-1, where it is hosted: https://health.aws.amazon.com/health/status At 09:15 UTC we started seeing signs of recovery. As of 09:30 UTC all high priority tasks were cleared which are those with the most visible user impact. Lower priority tasks were cleared by 10:20 UTC. AWS continue to work towards full resolution. We remain unable to reliably provision additional capacity in this data center. We have taken steps to ensure we retain the capacity we already have. Performance of the US data is now back to normal, we continue to monitor whilst the underlying AWS incident is active.

3. monitoring Oct 20, 2025, 01:40 PM UTC

   AWS continue to work towards a full resolution. We remain unable to reliably provision additional capacity in our US data center. However, since the previous message, we have managed to increase the available capacity. Performance of the US data center continues to be normal, and we will continue to monitor.

4. resolved Oct 20, 2025, 09:58 PM UTC

   On October 20th, AWS's us-east-1 region experienced a significant incident impacting multiple underlying services. This led to the operation of Cronofy's US data center, which is hosted in this AWS region, being severely impacted. None of Cronofy's other data centers were affected at any time. The main impact was felt between 07:15 and 09:15 UTC where many critical components struggled to communicate with AWS's services. Performance was severely degraded throughout this period. We saw some recovery from 09:15 UTC, with the backlog of high priority tasks completed by 09:30 UTC, these are the tasks most likely to have a user-noticeable impact. Some issues remained, mainly the ability to provision additional servers which has been widely reported by other AWS customers. However, we had sufficient capacity to clear the backlog of lower priority tasks by 10:20 UTC. Performance of the US data was back to normal from this time, and we continued to monitor. Throughout the day we ran with an altered configuration to obtain as much capacity as we could from AWS to provide as smooth a service as possible. Our ability to provision additional capacity was restored at 19:15 UTC. While the underlying AWS incident is still open, we are considering this incident to be resolved as we have been able to resume normal operations. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at [email protected].

5. postmortem Oct 22, 2025, 09:16 AM UTC

   ### Summary On October 20th, AWS's US-East-1 region experienced a significant incident impacting multiple underlying services. This led to the operation of Cronofy's US data center, which is hosted in this AWS region, being severely impacted. None of Cronofy's other data centers were affected at any time. The main impact was felt between 07:15 and 09:15 UTC where many critical components struggled to communicate with AWS's services. Performance was severely degraded throughout this period. We saw some recovery from 09:15 UTC, with the backlog of high priority tasks completed by 09:30 UTC, these are the tasks most likely to have a user-noticeable impact. Some issues remained, mainly the ability to provision additional servers which has been widely reported by other AWS customers. However, we had sufficient capacity to clear the backlog of lower priority tasks by 10:20 UTC. Performance of the US data was back to normal from this time, and we continued to monitor. Throughout the day we ran with an altered configuration to obtain as much capacity as we could from AWS to provide as smooth a service as possible. Our ability to provision additional capacity was restored at 19:15 UTC. The AWS incident was resolved at 22:53 and we consider this to be closed. ### Postmortem We always ask the questions: * Could this have been resolved sooner? * No, we very much at the mercy of AWS as they worked to resolve the issue and remove throttling as the services started to come back to life. The lack of access to the AWS console also made it difficult for our team to make meaningful changes to mitigate impact at the time. * As a business we operate our Data Centres completely independently of each other. This was a decision we made in order to ensure our customer’s data remains exactly where they choose to store it and removes any concerns around cross-region data transfers. As a result of this, we did not have the option to fallback onto another AWS region at the time of the US-East-1 outage, which could’ve provided increased availability at the height of the incident. We believe our initial decision to not perform any cross-region data transfers is still in the best interests of our customers, and don’t believe changing this to provide extra resiliency to be a viable option. * Could this have been identified sooner * No, our PagerDuty alerts came through 10 minutes before AWS announced the incident, and an internal incident channel was opened shortly afterwards. * Our platform, however, showed enough signs of life that our external monitoring services were not reporting downtime. * Our customers could have been informed sooner in this particular instance: * Our third-party StatusPage \([status.cronofy.com](http://status.cronofy.com)\) was not available during the worst of the incident, only returning at 09:50 UTC. Due to the AWS issues, we could also not update our DNS records to point to a different location. * We had to make use of improvised channels \(LinkedIn, X, Facebook\) until we could set up a temporary page at [status.cronofy.com](http://status.cronofy.com) at 08:45 UTC. This resulted in delays in informing our customers of the issues we were facing. * Could this have been prevented * No, this issue was due to an AWS incident and was not in our control. * The previously mentioned Data Residency choices mean that falling back to a different region was not an option. ### Actions * Review our external StatusPage to ensure we have the best setup for informing customers of ongoing incidents. * Review backup StatusPage options: * Update our internal Incident Playbook to list backup notification platform options * Review deployment pipeline setup for added resiliency in such situations

Read the full incident report →

Started Sep 15, 2025, 11:04 AM UTC· resolved Sep 15, 2025, 11:49 AM UTC

1. monitoring Sep 15, 2025, 11:04 AM UTC

   Since 10:40 UTC we have observed an increased error rate for Apple calendar sync operations. We are still seeing more successes than failures but consider there to be enough failures to mark Apple as degraded. Our retry mechanisms appear to be doing their job, but changes to calendars may take longer to sync both to and from Apple as a result of the underlying errors. We are monitoring the situation.

2. resolved Sep 15, 2025, 11:49 AM UTC

   Starting at 10:40 UTC we observed an elevated error rate when communicating with Apple's calendar service. Errors returned to regular levels by 11:35 UTC. Sync with Apple calendars may have taken longer both to and from Apple as a result of the underlying errors during this period.

Read the full incident report →

Started Aug 16, 2025, 03:36 PM UTC· resolved Aug 16, 2025, 04:28 PM UTC

1. investigating Aug 16, 2025, 03:36 PM UTC

   We are seeing a small number of 5XX errors being returned from our US data center. We are investigating and will update shortly.

2. identified Aug 16, 2025, 03:53 PM UTC

   Investigation has shown that between 15:19 and 15:27 UTC a database issue resulted in a partial loss of service. During this period requests may have returned a 5xx status and failed to process. The database issue has now been resolved and we're moving to monitor the environment.

3. monitoring Aug 16, 2025, 03:55 PM UTC

   We're monitoring the environment to ensure things continue to run as expected.

4. resolved Aug 16, 2025, 04:28 PM UTC

   Normal operations have continued and we are marking this incident as resolved. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at [email protected]

5. postmortem Aug 18, 2025, 04:33 PM UTC

   # Summary Between 15:19:38 UTC and 15:26:55 UTC on Saturday August 16th 2025, requests made to the API in our US data centre may have received a HTTP status of 500 - Internal Error. This was caused by a transaction on one of our database tables not releasing a lock in a timely fashion which, in turn, caused a bottleneck that led to service degradation as subsequent requests queued behind it. The root cause was identified during the incident and will be addressed by reviewing the way these transactions are handled with a view to preventing future locks. ## Timeline All times are UTC unless otherwise stated. 15:19 - Table lock begins 15:21 - On-call engineers alerted to issues processing requests as 5XX responses increase 15:22 - Initial investigation begins 15:27 - Database lock clears, backlog begins processing 15:28 - Wider environment confirmed to be functional. No undue load observed 15:30 - Lock on database table observed to be likely cause 15:32 - Lock on database table confirmed to be the cause 15:50 - We begin extended monitoring 16:28 - No further long-running locks observed. We mark the incident as resolved ## Retrospective We always ask the questions: * Could the issue have been resolved sooner? * Could be issue have been identified sooner? * Could be issue have been prevented? In this case, we feel that yes, we could have identified the root cause sooner as it has highlighted a gap in our database monitoring that we’ll be working to address. Similarly, it could have been resolved sooner if we were to opt to clear the lock ourselves. However, our preference is to avoid the need for intervention by addressing the way we approach these queries in the first place. ## Actions Our Site Reliability Engineers will be looking at addressing the pinholes in our monitoring of database transactions to improve visibility in the future, while we look to improve the approach we take to table locks and long running transactions.

Read the full incident report →

Started Jul 18, 2025, 03:23 PM UTC· resolved Jul 18, 2025, 04:55 PM UTC

1. investigating Jul 18, 2025, 03:23 PM UTC

   We've observed an elevated rate of errors from Google's Calendar API and are investigating the scale and impact. Updates to and from Google Calendar profiles may see slight delays during this time. We are monitoring the situation.

2. identified Jul 18, 2025, 03:39 PM UTC

   Google is returning a higher rate of 5xx errors when we try to perform token refresh operations. Approximately 1/3rd of our attempts to refresh tokens are seeing this error. The increase in errors started approximately 14:57 UTC. It appears our normal mechanisms are allowing retries to be processed successfully, and limiting the impact to the wider service. Updates to and from Google Calendar profiles may see slight delays during this time. Google appear to have acknowledged the incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9

3. monitoring Jul 18, 2025, 03:48 PM UTC

   Google's API error rate has dropped again and held at a normal level for the past 10 minutes, and both read and write operations to Google Calendar profiles appear to be back to our normal level of performance. We'll continue to monitor and confirm this holds out.

4. resolved Jul 18, 2025, 04:55 PM UTC

   API success rates have remained stable since our previous message, and Google have resolved their own incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9 Between the times of 14:57 UTC - 15:35 UTC, we saw elevated rates of errors, and Google Calendar users may have had slight delays to calendar reads and writes. Since 15:35 UTC everything has been functioning as normal.

Read the full incident report →

Started Jul 10, 2025, 09:45 AM UTC· resolved Jul 10, 2025, 06:58 PM UTC

1. investigating Jul 10, 2025, 09:45 AM UTC

   We've identified that some Outlook.com profiles may be increased error rates when authenticating with Microsoft's APIs, causing sync failures for those affected. This appears to be related to an ongoing issue with Outlook.com We're investigating the root cause and will update this incident as we understand more.

2. identified Jul 10, 2025, 10:14 AM UTC

   We've ascertained that the issue isn't down to changes on our side and are tracking live issue with Outlook.com. Microsoft are reporting on this as Issue ID EX1112414. We can see that not all profiles are affected and that, of those that are, retries do eventually succeed. We'll continue to monitor and advise of any further changes.

3. monitoring Jul 10, 2025, 11:05 AM UTC

   Error rates remain fairly consistent. We're continuing to monitor and will advise of any further changes.

4. monitoring Jul 10, 2025, 12:56 PM UTC

   Error rates remain fairly consistent. We're continuing to monitor and will advise of any further changes.

5. monitoring Jul 10, 2025, 03:49 PM UTC

   Error rates remain fairly consistent and Microsoft are in the process of rolling out a fix. We've yet to see the effects of this, are continuing to monitor, and will advise of any further changes.

6. monitoring Jul 10, 2025, 05:07 PM UTC

   As of 16:30 UTC we're seeing service levels back at their expected levels and a drastic reduction in error rates. Outlook.com profiles are now authenticating consistently. Microsoft Issue ID EX1112414 remains open however and so we'll continue to monitor things for a while longer in the expectation that this issue is now resolved but just to be sure.

7. resolved Jul 10, 2025, 06:58 PM UTC

   Microsoft Issue ID EX1112414 remains open however a preliminary root cause has been provided: "A recent service update to an authentication component is unintentionally preventing access for a subset of users, resulting in intermittent service unavailability." We've seen no further issues since our previous update and error rates remain consistent with healthy operations so we're going to consider this incident resolved.

Read the full incident report →

Started Jul 03, 2025, 02:36 PM UTC· resolved Jul 03, 2025, 04:07 PM UTC

1. investigating Jul 03, 2025, 02:36 PM UTC

   We are seeing a sudden increase in Google sync errors and are investigating.

2. monitoring Jul 03, 2025, 02:47 PM UTC

   The error rate has returned to normal. This does not align with any changes or operations on our side, and the error returned from Google suggests this may have been a temporary issue on their side. We are continuing to investigate to try and identify a root cause, as well as monitoring the situation for any further changes.

3. investigating Jul 03, 2025, 02:50 PM UTC

   We are seeing the issue recur and are continuing to investigate.

4. investigating Jul 03, 2025, 03:09 PM UTC

   Error rates have again returned to normal, and were affecting a fairly small number of operations spread across many accounts. We're continuing to investigate to try and find a pattern.

5. investigating Jul 03, 2025, 03:24 PM UTC

   The error rate has remained low, but we are still investigating and monitoring.

6. monitoring Jul 03, 2025, 03:36 PM UTC

   Error rates are still low and sync is working as expected. We are continuing to monitor the situation.

7. resolved Jul 03, 2025, 04:07 PM UTC

   Error rates have remained at normal levels and we believe the issue is over. After two increases in error rates, at 14:24 UTC and 14:45 UTC, Google sync has remained stable. A minority of sync operations failed and were retried, eventually completing successfully.

Read the full incident report →

Started Jun 12, 2025, 06:07 PM UTC· resolved Jun 12, 2025, 08:02 PM UTC

1. investigating Jun 12, 2025, 06:07 PM UTC

   We are seeing an elevated number of errors from Google and we are investigating

2. investigating Jun 12, 2025, 06:23 PM UTC

   Since 17:49 UTC we are receiving a greatly increased rate of API errors from Google. Many syncs are completing after retrying but customers may observe events taking slightly longer to appear in their calendars.

3. identified Jun 12, 2025, 06:46 PM UTC

   Google errors are remaining elevated. We have seen reports of other services also encountering issues with Google APIs and believe this to be related. We are continuing to investigate and monitor our services to be sure. We will update in ~30 minutes time if nothing substantial changes.

4. monitoring Jun 12, 2025, 06:58 PM UTC

   Google API calls have recovered and most are now succeeding. We are continuing to monitor the system to ensure this pattern continues. This issue is due to the following Google incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4

5. monitoring Jun 12, 2025, 07:30 PM UTC

   We continue to see healthy syncs with Google. However the Google incident remains open, citing that some areas have recovered but a full mitigation has not taken place yet. We are leaving this incident open to reflect that disruption may still be possible until Google confirm the issue is fully mitigated or resolved. Google Cloud incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4

6. resolved Jun 12, 2025, 08:02 PM UTC

   Google API calls began to error at abnormally high rates from 17:49 UTC. Upon investigation, we did not find a common factor or pattern behind the failures but kept investigating and engages some fallback mechanisms to keep the platform stable. We began to find other reports of Google API issues, ultimately having this confirmed by Google: - Google Workspace incident: https://www.google.com/appsstatus/dashboard/incidents/Eab7zGLWSaUJyh9c9R9L - Google Cloud incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4 We continued to monitor the system and confirmed that calendars were still syncing, albeit with occasional delays when syncs failed and had to be retried. At 18:48 UTC, API errors dropped drastically and returned to normal operational levels. We kept monitoring until we saw Google report the issue was resolved at 19:53 UTC.

Read the full incident report →

Started May 22, 2025, 01:08 PM UTC· resolved May 22, 2025, 01:31 PM UTC

1. identified May 22, 2025, 01:08 PM UTC

   We have identified a recent change has made public links error - we are reverting this imminently.

2. monitoring May 22, 2025, 01:25 PM UTC

   We have reverted the broken code, it is deploying at present and we are monitoring to find out how many requests may have been affected.

3. resolved May 22, 2025, 01:31 PM UTC

   We have reverted the change and confirmed that Public Links are now creating events as expected. A postmortem of the incident will take place and be attached to this incident the next 3 working days. If you have any queries in the interim, please contact us at [email protected]

4. postmortem May 23, 2025, 02:09 PM UTC

   Between 08:43 and 13:23 UTC on Thursday May 22nd 2025, attempts to book times via the Public Links feature of the Scheduler failed, visitors to Public Links would have instead seen an erroneous message that the Public Link was disabled. The root cause was the development of a new feature on top of Public Links, where normal booking flows were erroneously interpreted as making use of the new feature, and then failing a validation check that should not have been applied. As a result, 23 attempted bookings were not accepted. During the incident, we fixed the root cause with a patch. Following the incident, we contacted all affected owners of the impacted Public Links. ## Timeline All times are on May 22nd 2025. * 08:43 UTC - A change is merged which causes Public Link bookings to fail with a “disabled” message * 12:50 UTC - The issue is raised internally as a result of our own testing of the product * 12:57 UTC - The problematic code is identified, but we are unable to confidently issue a simple rollback since other changes to the area had been since introduced * 13:18 UTC - A fix to the code is written and reviewed * 13:23 UTC - The fix is approved and merged * 13:30-16:00 UTC - Work is undertaken to identify all affected public links * 16:29 UTC - All affected customers notified ## Retrospective We always ask the questions: * Could the issue have been resolved sooner? * Could be issue have been identified sooner? * Could be issue have been prevented? In this case, we feel our resolution time was reasonably good. In hindsight, we may have been able to have the fix released around 15 minutes faster had we acted more confidently to implement a patch concurrently with internal debate and investigation over the ability to roll back. We are not happy with our speed of identification, and also that this was an issue that could have been prevented quite easily. On identification, the issue was live for around 4 hours before being noticed internally. The failure in this case manifested in users being routed to an otherwise normal “disabled” page, and bookings not being made. It’s harder to alert on things _not_ happening, given the usage rate of this feature, and the natural peaks and troughs of daily activity. We did see an area for improvement in that we were using a normal “disabled” page as a catch-all for a few other error cases; by adding Telemetry around these different cases, we can positively identify unusual behaviour separately from the normal “disabled” case. We also saw possibilities to improve our playbook for monitoring usage of new features. In this case — had we configured a trigger with a shorter duration & period — we would have seen unexpected activity for the unreleased new feature. This would have led us to investigate and notice the issue sooner. Prevention is the place with the clearest room for improvement. We had focused our manual testing on the new feature being developed, and failed to test the vanilla case of Public Link bookings which were touched by the code changed. We use both automated and manual tests during feature development. The nature of the underlying issue, and the necessary interaction of multiple steps of the booking flow to cause it, made it less likely for our automated test suite to reasonably catch it. However, we failed to catch the erroneous code at Code Review stage, and failed to manually test the critical path adjacent to the changes being made. ## Actions We are going to more strictly manually test critical paths in our system when making changes adjacent to those areas to ensure there aren’t unintended side effects from in-development features. We are going to add increased monitoring of the error cases in the affected area — before they are sent to any fallback pages — so that anomalous activity and behaviour is positively identifiable and triggers a visible alert. We are going to review our playbook for new feature usage telemetry to add better guidance for engineers to set up triggers that are more visible by default.

Read the full incident report →

Started Apr 15, 2025, 06:32 PM UTC· resolved Apr 15, 2025, 08:22 PM UTC

1. investigating Apr 15, 2025, 06:32 PM UTC

   We are seeing a small number of 5XX errors being returned from our US data center. We are investigating and will update shortly.

2. investigating Apr 15, 2025, 06:56 PM UTC

   We have modified our configuration and seen a reduction in the number of errors to near zero. We are still investigating the cause of the issue.

3. monitoring Apr 15, 2025, 07:45 PM UTC

   We have identified the root cause of the issue. Normal operations have resumed. We will continue to monitor and will be carrying out further work to prevent a repeat.

4. resolved Apr 15, 2025, 08:22 PM UTC

   We have deployed a mitigation to prevent the same issue from arising. Normal operations have continued and we are marking this incident as resolved. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at [email protected]

5. postmortem Apr 17, 2025, 11:20 AM UTC

   Between 17:25 and 18:25 UTC on Tuesday April 16th 2025, some Availability Query requests were made that provoked an issue with the Availability calculation. This caused resource usage to balloon to the point where some servers ran out of memory. During this time a small number \(< 1%\) of requests to our US data center received an HTTP 500 response rather than being processed correctly. The root cause was an issue where some requests to the Availability Engine attempted to return values several orders of magnitude larger than intended, requiring significant resources to process over a period of 20 minutes. The subset of servers handling these requests eventually exhausted their available resources and began to fail. This lead all in-flight requests on those servers to receive a HTTP 500 status. Less than 1% of traffic was impacted between 17:25 - 18:25 UTC. During the incident, we identified and patched the root cause. We have also held a retrospective and decided on further actions to take to further harden these paths and monitoring to prevent similar issues and try to make them easier to spot. ## Timeline 17:25 UTC - A customer makes some Availability Queries that will go on to cause the memory exhaustion. 17:59 UTC - Several servers run out of memory and are killed. 18:03 UTC - On-call engineers are paged as the number of HTTP 5XX responses increases. 18:05 UTC - Engineers begin investigating, finding little that stands out as traffic and load patterns seem usual. 18:23 UTC - Engineers spot that a small number of servers have unusually high memory usage approaching the limit and are replaced. 18:31 UTC - Memory limits are temporarily increased to see if that eases pressure. It does not. 18:47 UTC - Customer stops making the Availability Query calls that are driving the issue. While normal service resumes quickly, we decide to keep the incident open until we understand and mitigate the root cause. 19:11 UTC - After investigating several areas of the infrastructure configuration, the Availability Query calls driving the memory consumption are spotted 19:57 UTC - A fix has been written and is reviewed 20:03 UTC - Fix is approved and merged 20:22 UTC - After confirming all systems are operating normally, the incident is resolved. ## Retrospective We always ask the questions: * Could the issue have been resolved sooner? * Could be issue have been identified sooner? * Could be issue have been prevented? In this case, we feel that our time to resolve and identify were fairly good given that a symptom of the issue was a small number of logs and metrics were missing due to the killed servers. Counter-intuitively, a larger scale issue would have been easier to spot, but this one was subtle enough that it took some time to uncover. Conversely, had the user who incidentally triggered the bug have been malicious, this could have lead to an effective denial of service attack on our system. We have reviewed the steps we would have taken and believe we would have been in a position to also mitigate this behavior, but may have an opportunity for clearer internal playbooks on the topic. When considering how long it took us to identify the root cause, it took some time to uncover that servers were being killed due to running our of memory. This is not a trivial thing to spot and record, but we are going to look at how we can capture and alert from this data. This would have saved us a few minutes of waiting and watching as the next generation of servers hit the same issue. Finally, prevention is the clearest step we can improve here. The issue was hard but not impossible to spot, and we are now much stricter on the values allowed in the impacted area of the system. We have also identified several other spots that our code can be more defensive to avoid similar incidents in the future. ## Actions Our Site Reliability Engineers will be looking at how we can track and monitor Out Of Memory errors, while our Product Engineers will be hardening the impacted parts of the Availability Engine to limit the scale of processing without impacting existing behavior. ## Further questions? If you have any further questions, please contact us at [[email protected]](mailto:[email protected])

Read the full incident report →

Started Feb 26, 2025, 08:02 AM UTC· resolved Feb 26, 2025, 02:29 PM UTC

1. identified Feb 26, 2025, 08:02 AM UTC

   We have identified an issue with calendar sync operations when connecting to Microsoft 365. We are seeing a higher error rate than usual, but most sync operations are still successful.

2. identified Feb 26, 2025, 08:39 AM UTC

   Our telemetry shows that the number of errors is decreasing but have not returned to normal levels, and that Microsoft Graph API syncs are operating as normal, this is only impacting EWS sync operations. Other background processing tasks are not impacted by this incident.

3. monitoring Feb 26, 2025, 09:14 AM UTC

   The number of EWS sync operation errors with Microsoft 365 continues to decrease, but they are not quite back to usual levels. We are continuing to monitor.

4. monitoring Feb 26, 2025, 09:44 AM UTC

   Metrics continue to trend in the right direction but we're still seeing a very small number of errors being returned from Microsoft 365. We are continuing to monitor.

5. monitoring Feb 26, 2025, 10:13 AM UTC

   We are continuing to monitor for any further issues.

6. monitoring Feb 26, 2025, 10:49 AM UTC

   We are continuing to monitor. The errors our telemetry shows are very gradually reducing. We will reduce the frequency of updates as we continue to monitor.

7. monitoring Feb 26, 2025, 11:23 AM UTC

   We strongly believe the errors we are seeing are related to ApplicationImpersonation being retired by Microsoft 365. For more information and to restore connections please follow the instructions on our docs site - https://docs.cronofy.com/calendar-admins/faqs/ews-migration/

8. monitoring Feb 26, 2025, 12:30 PM UTC

   We are continuing to monitor the situation and have no further updates at this time

9. monitoring Feb 26, 2025, 01:35 PM UTC

   We are continuing to monitor the situation. We are also sending additional emails to those who have yet to complete the migration from EWS to Graph API.

10. resolved Feb 26, 2025, 02:29 PM UTC

    If you have a service account connection through Microsoft 365 and have not completed the migration from Exchange Web Services (EWS) to Graph API then we will soon no longer be able to sync your calendars, and this may already be the case. Follow our guide to complete the migration as soon as possible, https://docs.cronofy.com/calendar-admins/faqs/ews-migration/ From 07:20 UTC today, we saw an increase in errors when performing EWS sync operations with Microsoft 365. The initial increase in errors tailed off as we stopped attempting to perform syncs which were no longer allowed. At 13:20 we began sending additional emails out to customers who have yet to complete the migration from EWS to Graph API. Again, we urge any customers who have not completed the migration from EWS to Graph API to do so as soon as possible.

Read the full incident report →

Started Jan 29, 2025, 10:32 AM UTC· resolved Jan 29, 2025, 11:51 AM UTC

1. investigating Jan 29, 2025, 10:32 AM UTC

   We're investigating an increase in API calls resulting in HTTP 500 errors

2. monitoring Jan 29, 2025, 10:38 AM UTC

   Error rates have returned to normal; we are continuing to monitor.

3. identified Jan 29, 2025, 11:03 AM UTC

   We are still seeing occasional errors via the API. These are in much less common and in smaller numbers, but not back to zero yet. The underlying cause is above average load on our database, which we are working to resolve.

4. monitoring Jan 29, 2025, 11:17 AM UTC

   Database load is back to normal and error have ceased. We are continuing to monitor to ensure normal operation has resumed.

5. resolved Jan 29, 2025, 11:51 AM UTC

   Normal operation has resumed and been consistent for the last 30 minutes, so this incident is resolved. About 1.75% of API calls between 10:08 and 10:38 received a HTTP 500 response due to database connections being refused. A combination of high-load tasks, both from API calls and internal DB processes, all happened at once. This caused the database to degrade in performance and refuse some connections. Attempts to manually kill some of the processes did not succeed, for reasons we will investigate. Once these jobs completed, performance recovered and normal operation resumed. Jobs which had failed and retried have since completed.

6. postmortem Feb 03, 2025, 06:26 PM UTC

   On Wednesday January 29th between 10:00 and 11:09 UTC our US data center experienced degraded performance. This was caused by multiple concurrent delete operations on a heavily used database table, which in turn resulted in slower than usual responses and some failures. The majority of API traffic was unaffected, but a small percentage of calls would have resulted in a HTTP 500 response. Further details, lessons learned, and further actions we will be taking can be found below. ## Timeline _All times rounded for clarity and UTC_ On Wednesday January 29th at 10:00 we began processing a large number of data deletion jobs in line with GDPR compliance. Due to the much higher than usual number of processes of this type, our database system soon began to struggle due to the number of deletions it was being asked to perform. In particular, the CPU usage was very high. At 10:08 the first alarm for high CPU usage alerted our engineering team, and they began to investigate the cause. Additional alarms followed notifying the team of failed jobs, 5xx responses and slower API response times. Starting at 10:25, the database was under so much load that some new connections were refused. Also, during this time our monitoring systems show our slowest API responses. This was cleared by 10:38. From 10:47 to 10:57 we saw response times increase again, but not as severely as during the earlier window. This time we didn’t see any refused database connections. From 11:00 activity on our database had reduced significantly and by 11:09 our database activity had returned to the usual levels. ## Retrospective The questions we ask ourselves in an incident retrospective are: * Could it have been identified sooner? * Could it have been resolved sooner? * Could it have been prevented? We look for holistic improvements alongside targeted ones. **Could it have been identified sooner?** No. We feel that during this incident we were quick to respond to the alarms we received and to identify the cause. **Could it have been resolved sooner?** Possibly. We were a little hesitant to completely stop the large number of processes from being performed as they related to compliance. However, we have identified improvements we can make to our incident playbook that could have helped to us to determine how many remaining tasks there were, as in other scenarios, direct intervention would have been required. **Could it have been prevented?** We use rate limits throughout Cronofy to provide a robust service. However, they were not applied in this area, which was also a resource intensive task. We have already applied rate limiting to spread the load out and prevent a repeat. ## Actions As mentioned, we have already applied rate limiting to the GDPR processes to spread out the load within sensible bounds. We are going to update our incident playbook to highlight where statistics around the remaining number of GDPR tasks can be found. Had this already been in place we would have been able to determine how much longer the incident was going to last. We’re also going to perform an audit of our other background jobs to determine whether there are other areas of our system that lack rate limits on the number of concurrent jobs. ## Further questions? If you have any further questions, please contact us at [[email protected]](mailto:[email protected])

Read the full incident report →

Started Dec 04, 2024, 04:12 PM UTC· resolved Dec 04, 2024, 08:30 PM UTC

1. identified Dec 04, 2024, 04:12 PM UTC

   We are working through an issue which is preventing our US data center from scaling as normal. Performance may be degraded as we approach our peak time (around 17:00 UTC) until this is resolved. This will primarily affect background processing times, with synchronization potentially taking longer than usual.

2. identified Dec 04, 2024, 05:23 PM UTC

   We are working through an issue which is preventing our US data center from scaling as normal. Performance of background processing was degrading somewhat through our peak time around 17:00 UTC with processing taking up to 60 seconds to be started rather than being near instant. As we are now past that peak, we anticipate any impact on background processing to be reduced but still possible until scaling operations return to normal.

3. identified Dec 04, 2024, 07:22 PM UTC

   We are working through an issue which is preventing our US data center from scaling as normal. Performance of background processing has been degraded somewhat through the past hour with processing taking up to 60 seconds to be started rather than being near instant. We are in contact with AWS to resolve the underlying problem.

4. monitoring Dec 04, 2024, 07:52 PM UTC

   A fix has been implemented in conjunction with AWS support and our ability to scale our resources has returned. Capacity has then been able to increase to match demand and service levels appear to have returned to normal. We are continuing to monitor to ensure service remains stable.

5. monitoring Dec 04, 2024, 08:30 PM UTC

   Our US data center experienced degraded performance today, primarily impacting background processing. There was an issue with the Amazon Elastic Kubernetes Service (EKS) cluster which affected our ability to scale capacity with demand. This was resolved by AWS intervening to resolve an issue with our EKS cluster. We will be receiving a RCA from AWS relating to this. Once we receive AWS's RCA, we will follow our own process to publish a postmortem of this incident. If you have any queries in the interim, please contact us at [email protected]

6. resolved Dec 04, 2024, 08:30 PM UTC

   Our US data center experienced degraded performance today, primarily impacting background processing. There was an issue with the Amazon Elastic Kubernetes Service (EKS) cluster which affected our ability to scale capacity with demand. This was resolved by AWS intervening to resolve an issue with our EKS cluster. We will be receiving a RCA from AWS relating to this. Once we receive AWS's RCA, we will follow our own process to publish a postmortem of this incident. If you have any queries in the interim, please contact us at [email protected]

7. postmortem Dec 17, 2024, 02:28 PM UTC

   On Wednesday December 4th between 13:30 and 19:40 UTC our US data center experienced a prolonged period of degraded performance primarily impacting background processing. This meant that operations such as synchronizing schedules which usually commence within seconds were delayed, at times, by a minute or more. This was caused by a degradation in service of an AWS managed service that is vital to our ability to scale our capacity to match demand. We are awaiting an RCA from AWS around this, but don't feel it is overly material to our own postmortem as we've been told we could not have resolved the underlying problem ourselves. _We will update this postmortem as necessary once the RCA has been received from AWS._ Further details, lessons learned, and further actions we will be taking can be found below. ## Timeline _All times rounded for clarity and UTC_ On Wednesday December 4th at 10:50 an additional permissions policy was added to the DE, US, and non-production Amazon Elastic Kubernetes Service \(EKS\) clusters in those environments. These three environments are older than the others, and so they lacked some permissions the newer environments had inherited by default. This was not affecting the operation of any of the data centers, but we wanted to bring their configuration inline after noticing the difference. These changes were applied successfully and everything appeared to operate as normal. More than two hours later at 13:20 we found the first signs of there being an issue within our US data center. _Without the RCA from AWS, we are assuming the configuration change is somehow related, but the fact that it only affected one of the three altered environments casts some doubt on that._ EKS provides the control plane of Kubernetes, with the nodes from the worker pool communicating with it to coordinate the distribution of work and scaling activity. At this time, the nodes and processes running on them stopped being able to communicate with the EKS as usual. This meant that things like processes responsible for triggering deployments to scale up could not do so, and that other processes that relied on obtaining leases from the control plane to elect a leader could not. Most crucially it meant that newly provisioned nodes could not fully join the cluster as they could not provision their networking stack and signal themselves as ready to run other processes. The first notification we received around the issue came at 13:40, and the first alert at 13:50. At 14:00 we attempted to increase the capacity of the background processes to provide headroom as things were not scaling dynamically but this was unsuccessful due to the underlying issue. At 14:15 we made the scaling change more directly to directly provision as much capacity as we could. We also attempted to add more compute capacity by adding more nodes to the cluster, but as they were unable to fully register themselves we were stuck with the capacity we had. For context, on the previous Wednesday, background processing fluctuated between 30 and 100 replicas during this period, with the servers in the cluster also fluctuating to provide the capacity for those replicas to run. As the issue began we were at 50 replicas, with direct intervention we were able to get it to 70 replicas. The gap between necessary capacity and peak capacity is the source of the performance degradation. We were able to process all tasks successfully but did not have the throughput available to keep up with spikes in load as they arrived. As we had made a change to multiple EKS clusters earlier in the day, we looked for signs for similar behavior in other environments, including those unchanged but did not find any. An incident being outside of our control, without it being part of a wider outage in a given AWS service or region, is historically rare. We spent the next two hours on activities such as undoing and redoing the change from earlier that day, manually comparing multiple environments configuration in case of some other drift, and such like. At 16:00 we came together to review the situation. As part of this we realized this may have become noticeable to users and that the situation was likely to become worse over the next hour as 17:00 is usually the time of peak load for our US data center. At 16:10 we opened this incident on our status page. We decided to try and add capacity to our US data center by provisioning a new EKS cluster and working out how to scale up capacity there. With our own diagnostic paths exhausted and our options going forward limited, we opened a ticket with AWS support at 16:55 whilst we working on provisioning a sibling cluster. At 17:25 AWS support request permission to review logs which we grant. Work continued to provision a new EKS cluster into which we could successfully register new nodes. Work then switched to how and what we would need to deploy into the second cluster to get something that would function without causing more issues than it would solve. At 18:45 we realized we had heard nothing from AWS for over an hour and so initiated a chat session. After some back and forth we had confirmation that it was being investigated at 19:09. At 19:13 we were asked to check how the cluster looked and there were signs of improvement but still issues. At 19:26 the AWS agent joined our conference call and helped us triage lingering issues. At 19:30 we'd been able to add additional nodes to the cluster which meant we could deal with the background processing backlog. By 19:35 all issues within the cluster had been resolved and it further scaled up through the automatic mechanisms. We continued to monitor whilst reverting changes made to provide as good as service as was possible throughout the incident, before returning fully to our usual configuration around an hour later. ## Retrospective The questions we ask ourselves in an incident retrospective are: * Could it have been identified sooner? * Could it have been resolved sooner? * Could it have been prevented? Also, we don't want to focus too heavily on this specifics of an individual incident, instead look for holistic improvements alongside targeted ones. **Could it have been identified sooner?** Yes, whilst we received alerts they were slower than we would like and pointing towards more symptoms of the issue rather than the issue itself. **Could it have been resolved sooner?** Absolutely, with AWS having to resolve it, opening a ticket with them much sooner would have helped. We also suspect opening a chat with them rather than an email ticket may have influenced the speed of response. **Could it have been prevented?** From our actions, we don’t believe so. The bargain you make with managed services is that if you use them correctly, they’ll work. To our current knowledge, AWS failed on their side of the bargain which we can’t prevent without extremely significant overhead day-to-day. ## Actions This incident uncovered a flavor of infrastructure failure that is not sufficient covered by our alerting. We’ll be reviewing and improving alerts within this area of our stack to guide our future selves more rapidly to the root cause of similar issues in future. On a similar note, we found our diagnostic tools to be weak in this area, and relied on ad-hoc knowledge more than we would like. In concert with the review of alerts, we’ll improve our playbooks and scripts for assessing the situation when such alerts are triggered. Finally, we are documenting guidance around when and how support tickets should be raised with AWS to reduce the amount of ad-hoc decisions we have to make on this front. ## Further questions? If you have any further questions, please contact us at [[email protected]](mailto:[email protected])

Read the full incident report →

Started Nov 25, 2024, 04:27 PM UTC· resolved Nov 25, 2024, 10:27 PM UTC

1. identified Nov 25, 2024, 04:27 PM UTC

   We have been tracking a degradation in MS Graph connections since approximately 07:30 UTC today. While most operations have been succeeding in short order, this has worsened to the point where sync is starting to be noticeably degraded for some accounts. Microsoft are reporting on this as incident "MO941162". We are continuing to monitor and mitigate where we can, and will provide another update at 17:30 UTC.

2. identified Nov 25, 2024, 05:31 PM UTC

   Many Graph API calls to Microsoft 365 are still timing out and we have yet to see any improvement. We are continuing to monitor the situation and mitigate any impact on operations. We will update again at 18:30 UTC.

3. monitoring Nov 25, 2024, 06:27 PM UTC

   Many calls to Microsoft 365 are still failing. We have seen some improvement over the past hour but are still far from a full recovery. Synchronization of affected calendars will be degraded with failures retried automatically as required. Microsoft are reporting on this as incident MO941162, with fixes in progress but currently no timeline to resolution. We are continuing to monitor and mitigate where we can, and will provide another update by 19:30 UTC.

4. monitoring Nov 25, 2024, 07:29 PM UTC

   We have seen little further improvement of reliability of calls to Microsoft 365. We are continuing to monitor our systems and Microsoft incident MO941162. Sync tasks continue to retry automatically. We will provide a further update by 20:30 UTC.

5. monitoring Nov 25, 2024, 08:26 PM UTC

   We have seen little improvement of reliability of calls to Microsoft 365. However, we are now well past the busiest point of the day for Cronofy's services which means we're in a more stable position overall. We are continuing to monitor our systems and Microsoft incident MO941162. Microsoft are continuing to work on the underlying issue and sync tasks continue to retry automatically. We will provide a further update by 21:30 UTC.

6. monitoring Nov 25, 2024, 09:27 PM UTC

   We have seen some improvement in the reliability of calls to Microsoft 365, but are far from normal service. We remain past the busiest point of the day for Cronofy's services which means we're in a more stable position overall. We are continuing to monitor our systems and Microsoft incident MO941162. Microsoft are continuing to work on the underlying issue and sync tasks continue to retry automatically. We will provide a further update by 22:30 UTC.

7. resolved Nov 25, 2024, 10:27 PM UTC

   We have seen a significant improvement in the reliability of calls to Microsoft 365. This has now been close to, or within, the bounds of usual behavior for around an hour. This aligns with Microsoft's communication against incident MO941162. Microsoft are not saying it is fully resolved but that they have mitigations in place, and are continuing work to fully resolve the problem. As the impact of this has been minimal for a significant period, we are resolving this incident. We will keep a closer eye on this area until Microsoft call their incident resolved as well.

Read the full incident report →