Coveralls Outage History

User in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST. We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next 45min - 1hr.

The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here.

A fix has been implemented and we are monitoring the results.

Operational in all regions. We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

This was the second wave of the attack we experienced this weekend, broadened and tripled in volume. We implemented a second phase of defense which appear to be working and will continue monitoring for the next hour.

All systems fully functional. We have identified increased latency for large repos again and have proactively arrested sources of excess requests. We expect to restore normal latency within the hour.

We have reduced traffic in affected queues by 75%. We still expect to clear the queue within about 1 hr. We will post updates here. If you think you may have been paused as a user with a large repo (>=5K source files) that has uploaded 3-10x normal daily uploads, please reach out to us at [email protected] and we'll confirm and give you some next steps to avoid further pauses (see Usage Add-Ons at https://coveralls.io/pricing).

We are continuing to monitor for any further issues.

We are continuing to proactively monitor and clear the large repo queues. We are working on a longer term solution over the weekend and will post about it in the postmortem for this issue when complete. We will keep this incident open as long as we have large repo queues with above-average latency.

Latency remains elevated by about 30% for large repos. We are continuing to monitor and do what we can to mitigate until we can implement a longer-term solution.

We are continuing to monitor as we scale to clear backlogged jobs.

We received reports from EU customers of elevated latency and have resolved the cause. We will be managing workloads to ensure standard processing times for all new builds, and scaling resources to clear delayed builds. Follow this incident for updates.

Build times are returning to normal for all builds created since 05:00AM PDT. Older builds are being processed by scaled resources and should clear in 45min to 1 hr. If you are having issues with a slow or stuck build, especially if priority, feel free to reach out to us at [email protected]. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number.

We received reports of 504 Gateway Timeouts from our Web Servers (coverage report uploads) between 6:00am-6:20am PDT and several moments ago between 11:15am-11:30am PDT. We have recovered service to all web servers and are monitoring for further occurrences.

We have received some reports of 504 errors this morning. We are monitoring to avoid further occurrences. We are looking into any gaps in our new monitoring regime, which has drastically reduced occurrences, to see where we can further improve.

We've received results of 504 Timeout errors from 7am PDT through 8:15am PDT. We have addressed the issue and issue was resolved as 8:19am PDT. We are monitoring for further issues.

Based on the details of today's incident, we've added another layer of monitoring that should give us earlier indications of this error state and allow us to respond more quickly.