Close experienced a major incident on February 11, 2025 affecting Application UI and Syncing and 1 more component, lasting 1h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 11, 2025, 10:31 PM UTC
We have become aware of degraded performance of the Close system. We are actively triaging the issue. Updates will be posted as they become available.
- identified Feb 11, 2025, 10:40 PM UTC
The issue has been identified. A fix is being rolled out to our production system.
- monitoring Feb 11, 2025, 10:47 PM UTC
A fix has been implemented. We are monitoring the results.
- monitoring Feb 11, 2025, 10:58 PM UTC
We are continuing to monitor for any further issues.
- monitoring Feb 11, 2025, 11:20 PM UTC
Our App UI and API have returned to normal performance. Our email and calendar syncing systems are operational but delayed. We are continuing to monitor for further issues.
- resolved Feb 11, 2025, 11:34 PM UTC
This incident has been resolved.
- postmortem Feb 14, 2025, 07:23 PM UTC
Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. ## Impact Beginning 22:00 UTC until 23:03 UTC on Tuesday February 11, 2025 users of the Close App experienced significantly degraded performance and an elevated error rate. This included slow page loads, long API response times, delayed email syncing, and delayed calendar syncing for all customers. ## Root Cause and Resolution At 22:00 UTC on Tuesday February 11, 2025 a change was deployed to the production environment of the Close system. This change introduced a poorly optimized database query. This caused our primary back end database to become overloaded and unstable. Close Engineering identified and removed the offending database query by 22:47 UTC. Performance began to gradually return to baseline, fully recovering by 23:03 UTC. ## Timeline * 22:00 UTC - A change is deployed to production introducing a poorly optimized database query * 22:01 UTC - Load on our primary back end database begins to climb * 22:06 UTC - Load on our primary back end database breaches critical thresholds alerting Close Engineering * 22:17 UTC - Close Engineering isolates the poorly optimized database query and begins to remove it from production * 22:47 UTC - Removal of the poorly optimized database query completes * 23:03 UTC - Close system performance returns to baseline