Zomentum experienced a critical incident on December 18, 2023 affecting Main API server, lasting 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 18, 2023, 04:06 AM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- resolved Dec 18, 2023, 04:18 AM UTC
This incident has been resolved.
- postmortem Dec 18, 2023, 04:23 AM UTC
#### Incident Overview We experienced a significant service interruption with Zomentum Grow, primarily impacting our API functionalities. This incident was due to the unintentional deletion of the main server's underlying image, a result of a misconfigured lifecycle policy. #### Incident Timeline and Impact * **Dec 18, 04:06 UTC:** The main server hosting most of Zomentum Grow's APIs became unavailable. * **Dec 18, 04:07 UTC:** Our monitoring systems detected the downtime, and our team was immediately alerted. * **Dec 18, 04:18 UTC:** The service was affected for 12 minutes, during which users experienced disrupted access to our APIs. #### Resolution and Recovery * **Dec 18, 04:12 UTC:** We successfully restored the server to its healthy status. * **Root Cause:** The root cause was identified as an erroneous deletion of the server image due to a misconfigured lifecycle policy. * **Corrective Measures:** Our team promptly rectified the issue and has since revised the lifecycle policy to prevent similar incidents in the future. #### Moving Forward We have taken this incident very seriously and implemented additional safeguards to ensure the stability and reliability of our services. We understand the critical nature of our service to your operations and apologize for any inconvenience caused. For ongoing updates and more information, please visit our https://status.zomentum.com or contact us at [email protected]. We appreciate your understanding and support.