Semaphore had 27 outages in the last 2 years totaling 76h 59m of downtime — averaging 1.1 incidents per month.
There were 27 Semaphore outages since December 23, 2024 totaling 76h 59m of downtime. Each is summarised below — incident details, duration, and resolution information.
We're experiencing an elevated level of 500's errors and are currently looking into the issue.
The issue has been identified, and our team is implementing a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We identified slow communication between Semaphore jobs and GitHub. We are continuing to work with our upstream provider to resolve this. For impacted jobs, please replace the normal checkout command with the following: retry --times 5 "timeout 40 bash -lc 'source ~/.toolbox/toolbox; checkout'" And afterwards, use cd to change the directory to the cloned repository (replace with your project).
A fix has been implemented and we are monitoring the results.
We identified slow GitHub operations related to problems with the network, and we deployed a fix to re-route those operations to a healthier server. Some HTTPS operations started to fail because of that. To disable the fix, you should use the command: sudo sed -i 's/^\([0-9].*github\.com\)/#\1/' /etc/hosts And to re-enable, you can use: sudo sed -i 's/^#\([0-9].*github\.com\)/\1/' /etc/hosts
A fix has been implemented, and everything should be back to normal. The previously shared snippets can be removed.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
We are continuing to investigate job start failures and processing delays. We will continue to update while investigating this.
The issue has been identified and we are working on a fix. We will provide our next update within 30 minutes or sooner if things change. We thank you for your patience while we continue to work to resolve this issue.
A fix has been implemented and we are monitoring the results. We will provide our next update within 30 minutes or sooner if things change.
This incident has been resolved.
Webhook delivery delays from GitHub led to a backlog of workflows being triggered all at once as delivery resumes. This caused a temporary spike in demand for some Semaphore machines. We are actively working to reduce the impact and restore normal capacity. We continue monitoring both GitHub’s incident and Semaphore infrastructure closely, and we will provide further updates as conditions stabilize.
We’re seeing considerable improvement in internal performance metrics, and we will continue monitoring until conditions fully stabilize.
This incident has been resolved.
We are currently investigating this issue.
Webhook delivery delays between GitHub and Semaphore appear to be caused by degraded performance on GitHub’s side. GitHub is currently reporting an active incident that appears related. We are monitoring the situation closely and will provide updates as GitHub resolves the issue.
GitHub has posted an update indicating recovery of the aforementioned issue. We continue monitoring Semaphore closely as things stabilize. (GitHub Status: https://www.githubstatus.com/incidents/f314nlctbfs5)
This incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating the issue. Implementing the retry command should help mitigate the issue: https://docs.semaphore.io/reference/toolbox#retry
We are continuing to investigate this issue.
We’ve identified that these issues are being caused by inconsistent network routing between Europe and the United States on upstream providers. Semaphore’s systems remain healthy, but affected routes can cause slower file transfer and occasional connection timeouts.
The connection seems stable and we keep monitoring it.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified, and we are working on a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing investigating this issue.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
We're finalizing a fix for this issue.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
Semaphore is now operating normally. We’re currently monitoring the situation to ensure continued stability.
This incident has been resolved.
Some users may see incorrect account or billing messages in the UI. CI pipelines might be affected. We're investigating and will provide updates shortly.
A fix has been implemented and we are monitoring the results. Some users may still see outdated messages due to caching, but functionality is unaffected.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
We are continuing to work on a fix for this issue.
Job delays on the Ubuntu Noble (24.04) image.
A fix was deployed for e2 machines. We are continuing work on f1 machines.
We have deployed a fix for f1 machines, and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
Due to several issues reported related to the latest Chrome versions, we are in the process of a rollback of the Ubuntu image update.
The rollback of the Ubuntu image has been completed and the issue has been resolved.
We experienced an outage affecting a subset of API endpoints due to a misconfiguration in API request routing. The issue persisted for approximately 2 hours and 30 minutes and was fully resolved at 16:30 UTC.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.