Coveralls Outage History

This incident has been resolved.

Just a note to address the gap in our Coverage Calculation Background Job Dequeue Time Graph today, MON, FEB 2 from 00:15:00 PST to 07:35:00 PST: Coveralls experienced no disruption in service at this time. Instead, a deployment issue cause the cron job that reports the metric to fail until it was resolved at 07:35:00 PST.

We had to pause some queues to recover performance for new jobs, but will clear those as soon as we've recovered normal build times for new jobs (ETA: 15-min). If you are a user in APAC or EU, you may have had your jobs paused. One repo in particular has represented 90% overnight workload. We will reach out to that user.

This incident has been resolved. Build times for all new builds is normal across the board. We are still clearing a backlog of background jobs, which should be clear in the next 15-20-min. If you are having issues with a slow or stuck build, feel free to reach out to us at [email protected]. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number.

We are closing this incident having received no further reports of 504 errors today. We will continue to monitor for them.

We have received no further reports of 504 timeout errors on coverage report uploads today, but we continue to monitor and will continue trying to improve our mitigations.

This incident is resolved. We have applied an additional layer of monitoring that should help us catch these cases earlier.

We are closing this incident after recent mitigations and a weekend without any reports. We continue to implement mitigations and infrastructure changes we believe will further reduce incidents of this error type.

Closing this incident as a potential false alarm. Connection stats show no issues and we cannot reproduce any responses of, "This website is under heavy load (queue full)." Nor do we see any in logs from this morning. We now believe the report we received this morning may have actually been encountered yesterday, around the same time as another report, before the incident was resolved. (Confirmed.) Please let us know if you have received the Web or API response, "This website is under heavy load (queue full)," and when you encountered it—which would be especially helpful.

We experienced a period of elevated latency due to overnight traffic for about 6 hours today, Fri, Aug 8, from about 1am-7am PDT. We paused a couple of outlier repos, scaled servers to clear a backlog, and resolved ongoing latency. It took approximately 30-min to clear our remaining backlog of background jobs. We'd also like to take this opportunity to address a question we get, which is: Q: "If the Dequeue Time Graph is missing from the Status Page, does that mean processing has stopped?" A: No. While a missing graph can indicate a problem, it’s often isolated to the sampling process itself—either the cron job that collects the data or the single server we currently use for sampling. This “flagship” server runs additional packages and handles control and reporting tasks beyond normal app serving. At times, those extra duties can overload it, causing sampling failures without affecting overall build processing. We plan to broaden our sampling to multiple servers in the future.

This incident has been resolved.

All builds from today (Jun 3) have been processed. As background job queues cleared, build times returned to normal. We will continue monitoring.