Is Coveralls down?

All systems fully functional. We have identified increased latency for large repos again and have proactively arrested sources of excess requests. We expect to restore normal latency within the hour.

We have reduced traffic in affected queues by 75%. We still expect to clear the queue within about 1 hr. We will post updates here. If you think you may have been paused as a user with a large repo (>=5K source files) that has uploaded 3-10x normal daily uploads, please reach out to us at [email protected] and we'll confirm and give you some next steps to avoid further pauses (see Usage Add-Ons at https://coveralls.io/pricing).

We are continuing to monitor for any further issues.

We are continuing to proactively monitor and clear the large repo queues. We are working on a longer term solution over the weekend and will post about it in the postmortem for this issue when complete. We will keep this incident open as long as we have large repo queues with above-average latency.

Latency remains elevated by about 30% for large repos. We are continuing to monitor and do what we can to mitigate until we can implement a longer-term solution.

This issue was resolved over the weekend. We will continue to monitor for elevated latency in the modified queues.

We are monitoring increased latency for larger repos (5K+ source files) due to elevated traffic in background job queues. We will pause outlier repos (more than 2x average traffic) to allow queues to clear for the general population, then restore once queues have cleared. If you think your repo may have been one of those paused, two things: 1) Reach out to us at [email protected] and we'll confirm; and 2) Consider purchasing a usage add-on so that your jobs are not throttled for fair-use, or an enterprise plan with isolated infrastructure for maximum performance: https://coveralls.io/pricing

We are continuing to monitor this situation. Latency is much reduced but still elevated. We will close this incident when it's fully restored to normal.

This incident has been resolved but we believe it triggered a worsened incident overnight Sun night/Mon morning (US PDT) which has just been resolved. To be confirmed by full RCA, we believe a deluge of large repo uploads tied up individual web servers that handle frontline requests. Each web server is able to recover on its own, and did, but as volume increased all servers were eventually affected, only allowing short windows where requests could get through—rejecting most requests with 504 errors. We'll post a post-mortem when we understand more about what happened and how to prevent it going forward.

We are seeing increased latency for larger repos (5K source files and greater). While the site is fully operational, you may experience increased delays before receiving finished build pages and coverage notifications to your PRs if your repo had >=5K source files. We are monitoring and will pause any outlier repos (repos with unexplained, excessive usage) to allow general traffic in large repo queues to clear.

We've had too offload some outlier repos to let the general public of users with heavy repos to clear. There were three such repos. We will restart processing on those repos as soon as general queues have cleared. If you think you may be one of the repos with excessive traffic that was paused today, just reach out and we'll let you know: [email protected]. We expect to clear the general queues by 6pm PDT. After that we will restore processing for outlier and expect that to complete by this evening between 10p-12a PDT.

Increased latency for large repos has been resolved for the general public. We will be clearing three outlier repos overnight, which should be fully cleared by tomorrow AM.

We have received reports of increased report latency for larger projects (5K file and up). We are doing our best to clear the background job queues for these projects. If you have one of these projects, feel free to reach out and we'll check status for you.

A fix has been implemented and we are monitoring the results.

We are still receiving reports of latency for large projects and are seeing backlogs of background jobs in queues dedicated to large repos (5K+ source files). We may pause some outlier repos with excessive numbers of jobs in order to clear general traffic. If you believe you may have been paused, reach out and we'll let you know.

We are keeping this incident open as we continue monitoring.

We have moved all repos with more than 300 jobs in queue to dedicated queues for processing. These 7 repos were responsible for 75% traffic in our heavy repos queue. This move allows us to provide normal processing times for non-outlier repos and dedicated resources to outlier repos. Outlier repos by their job volume will still take longer to clear. If you believe you may be one of these outlier repos, please reach out and we'll confirm and give you an idea of when your jobs will be processed. If you'd like faster processing, we can also pause processing on older job, for instance, anything older than 1 hour (or 2 hours, or 3 hours) ago.

We continue to monitor background processing queues for larger repos (5K+ files), as we continue offloading incoming jobs from the outlier repos whose volume led to the original backlog. If you think you may have been paused as one of these repos with outlier-level volume, please reach out to us at [email protected] and we'll confirm status and get you an ETA on when your remaining jobs will be processed.

We are continuing to monitor for any further issues.

We've cleared the backlog of background jobs for larger projects from our heavy project queues, with the exception of nine projects whose volume continues to exceed our daily fair use thresholds. Those nine have been redirected to secondary and tertiary queues that drain only after the primary queue is clear. Most of their jobs have now been processed, but a few projects generate more jobs than a normal day + overnight cycle can drain, so some backlog remains. We're continuing to work those queues down manually. If you think one of your projects may be affected, reach out to us at [email protected] and we'll check status for you.

We are closing this incident. Our main background processing queue for larger repos has had no backups in 48 hrs. We will continue monitoring for recurrence.

We are currently investigating an issue with an apparently expired SSL cert. Our orchestration layer shows no issue with certs so we need to dig deeper.

This incident has been resolved.