Close experienced a notice incident on May 24, 2024 affecting Application UI and API, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved May 24, 2024, 11:13 AM UTC
API endpoints and application areas backed by Search infrastructure - leads, contacts, opportunities list pages, and certain reporting pages were inaccessible for 20 minutes. This has been resolved.
- postmortem May 24, 2024, 12:06 PM UTC
Close sincerely apologizes for the interruption of our service. We take the stability of our platform very seriously. Below is an explanation of what happened and how we will prevent another such interruption from occurring. ## Impact API endpoints and application areas backed by Search infrastructure - leads, contacts, opportunities list pages, and certain reporting pages were inaccessible for 20 minutes. ## Root Cause and Resolution As part of an ongoing effort to increase security of Close application, we shipped a change to how authentication is handled between the application and internal search clusters. However, a certain kind of authentication exchange wasn’t handled correctly, which caused most of the requests to Search infrastructure to fail in production. We have reverted the change, and will be improving our testing setup to catch these issues during development, as well as deployment and monitoring systems to catch production issues earlier. ## Timeline * 10:20 UTC - An engineer rolls out an incorrect change in how authentication handled between application and internal search clusters. * 10:21 UTC - Monitoring systems alert on-call engineer * 10:33 UTC - The change was reverted * 10:40 UTC - The application functionality fully recovered