Coveralls experienced a minor incident on October 18, 2023 affecting Coveralls.io Web, lasting 1h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 18, 2023, 10:52 PM UTC
We've received reports of slow build times from some users associated with some repos. We have identified one server in OOM hang that is servicing one queue, so customers in this queue are most likely the ones affected. We will be clearing that queue as quickly as possible. If you are experiencing slow builds, feel free to reach out to [email protected] and we can find out if any of your recent builds are involved.
- monitoring Oct 19, 2023, 12:04 AM UTC
We are continuing to clear jobs from the backed-up queue and are at about 80%. We'll continue to monitor and update, but we expect most stalled jobs to start processing again in the next 15-min.
- resolved Oct 19, 2023, 12:25 AM UTC
The single backed-up queue on our troubled server is now clear, so all jobs related to slow builds are no longer pending, but are now in-progress (being processed by Coveralls). Affected builds aside, performance should be back to normal for all users. Again, if you think you were affected by, or are still affected by this incident, please reach out to [email protected] with your build URLs and we'll look into them for you.