Cronofy incident

Increased API errors in our US data center

Minor Resolved View vendor source →

Cronofy experienced a minor incident on November 30, 2025, lasting 1h 44m. The incident has been resolved; the full update timeline is below.

Started
Nov 30, 2025, 08:51 PM UTC
Resolved
Nov 30, 2025, 10:35 PM UTC
Duration
1h 44m
Detected by Pingoru
Nov 30, 2025, 08:51 PM UTC

Update timeline

  1. monitoring Nov 30, 2025, 08:51 PM UTC

    We have chosen to initiate a database failover in our US data center. Because of this, customers may have received a 500 Internal Server error between 20:45:15 and 20:47:30. The vast majority of these errors were observed between 20:45:15 and 20:46:00. We are now monitoring the database post-failover.

  2. monitoring Nov 30, 2025, 09:11 PM UTC

    We're continuing to monitor the situation. An additional spike in server errors was observed between 21:03:15 and 21:03:30

  3. monitoring Nov 30, 2025, 09:57 PM UTC

    We are continuing to monitor post-failover

  4. resolved Nov 30, 2025, 10:35 PM UTC

    Between 20:45:15 and 20:47:30, and 21:03:15 and 21:03:30 UTC, we observed increased API error rates in our US data center. The first window of these was caused by a database failover, the second of these was caused by a reboot of the database cluster reader node. We took this action in an attempt to alleviate a small but persistent number of database errors that we had seen since the earlier maintenance. During the maintenance, we believed these errors were transient and that they would dissipate throughout the day. As this did not happen, we felt it was necessary to take action now, and that action was to failover, reboot, and then ultimately replace the affected node. Since the replacement, we have not observed the aforementioned database errors. If you have any questions, please email [email protected].