Codefresh Outage History

**Impact:** The marketing website for Codefresh, `https://codefresh.io/`was unavailable globally for around 6 hours. **Root Cause:** A configuration issue led to a traffic routing problem. Because of the nature of intermediary responses, the routing loop was not immediately diagnosed. **Resolution:** The routing configuration was restored. The site is now fully accessible. **Detection:** The issue was identified internally through monitoring and also reported by our team.

**Impact:** Following changes made during recent scheduled maintenance, some customers experienced slower load times in the Codefresh SAAS platform UI. Builds were unaffected. **Detection:** Our monitoring systems alerted us to unusual activity, and our team quickly initiated an investigation. Customer reports of slower performance further confirmed the issue, allowing us to prioritize and address it promptly. **Root Cause:** The traffic imbalance was due to a combination of technical configurations that resulted in uneven resource allocation across our zones immediately after the maintenance. This led to one zone handling a disproportionate amount of traffic, impacting website responsiveness for some users. **Resolution:** Our team implemented several measures to redistribute traffic evenly across all zones. These adjustments restored balanced performance, with monitoring systems ensuring stability.

**Impact**: Some accounts sporadically experienced longer pending times than usual on a portion of their builds for a day. **Detection**: Issue was reported by a customer, and shortly after confirmed by Codefresh’s platform monitoring alerts. **Root Cause**: This issue was caused by a bug in MongoDB driver. The MongoDB driver was upgraded in Codefresh services as part of our efforts to improve performance, but this version contained a bug that caused Mongoose queries to hang when under heavy load without returning or throwing errors. This resulted in the Codefresh build manager randomly getting stuck when enough queries were hanging under certain conditions. **Resolution**: A temporary solution to improve build queries queue behavior was initially implemented to alleviate the issue for affected customers. The actual root cause was identified the following week, and the issue was resolved by downgrading the MongoDB driver to a version that did not contain the bug.

**Impact**: We had a 10 hybrid runners \(no more than 10\) that were unable to communicate with our API for a day, and therefore were unable to fetch and run pipelines. **Detection**: We were informed of this issue by customers. **Root Cause**: We identified an issue with our certificate rotation which failed to generate new certificates as required for this subset of runners. **Resolution**: We were able to resolve the issue by manually recreating the certificates required, which were then updated to the runners on the next build, restoring the service for all impacted customers. Further mitigation was done to ensure the issue with certificate rotation was also rectified. We are working on monitoring improvements in this area