Green Security experienced a major incident on April 3, 2025 affecting Vendor Credentialing Portal, lasting 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 03, 2025, 07:29 PM UTC
We are currently investigating the issue and hope to have it resolved quickly.
- identified Apr 03, 2025, 07:31 PM UTC
The issue has been identified and our operations team is working to restore the full functionality of the site.
- monitoring Apr 03, 2025, 07:40 PM UTC
A fix has been implemented and we are monitoring the system as it comes back online.
- resolved Apr 03, 2025, 08:16 PM UTC
This incident has been resolved.
- postmortem Apr 03, 2025, 08:30 PM UTC
We sincerely apologize for the inconvenience caused by the recent outage that lasted for 25 minutes. During this time, you may have experienced elevated errors and were unable to access the Green Security Platform to check in vendors. We understand how critical this service is for your operations, and we deeply regret any disruption this may have caused. The root cause of the outage was identified as a manual backup of our database that was initiated as part of a special migration. While this backup was running, a routine deployment attempted to modify the tables in the database. This conflict caused the system to become unresponsive until our operators identified the issue and unblocked the deployment. To prevent such incidents in the future, we have put guards in place to prevent ad-hoc database backups and deployments from running at the same time. Additionally, we have implemented immediate fixes and are working on long-term strategies to enhance the stability and reliability of our services. Our team is dedicated to ensuring that our platform remains accessible and secure at all times. Thank you for your patience and understanding. We are committed to providing you with the best possible service and will continue to improve our systems to meet your needs.