Coveralls Outage History

Started Feb 26, 2025, 03:57 PM UTC· resolved Feb 26, 2025, 05:40 PM UTC

1. investigating Feb 26, 2025, 03:57 PM UTC

   We are currently investigating this issue.

2. identified Feb 26, 2025, 04:30 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 26, 2025, 04:40 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Feb 26, 2025, 05:21 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Feb 26, 2025, 05:40 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 20, 2025, 07:45 PM UTC· resolved Feb 21, 2025, 08:15 AM UTC

1. investigating Feb 20, 2025, 07:45 PM UTC

   A number of users with repos that have parallel builds have reported encountering the following error on their Build Pages: Error message: DataTables warning: table id=DataTables_Table_0 - Ajax error. For more information about this error, please see http://datatables.net/tn/7 This error indicates an issue rendering elements of the Source Files table in your Build Page. Usually the TREE View of that table. It sometimes appears for a short time before the page is fully rendered, but if it persists it could indicate that Coveralls had a problem successfully aggregating all of the source files for your parallel build.

2. identified Feb 20, 2025, 08:53 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Feb 20, 2025, 10:26 PM UTC

   We are actively working on a fix for this issue. Based on our current timeline, we expect to deploy the fix later today, closer to the end of the US Pacific Time (PST) workday. For most US users, this means the issue should be resolved by tomorrow morning. Customers in the EU and Asia may experience the rollout during their working hours. We appreciate your patience as we complete this process.

4. monitoring Feb 21, 2025, 03:09 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 21, 2025, 08:15 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 12, 2025, 08:54 PM UTC· resolved Feb 13, 2025, 08:41 PM UTC

1. investigating Feb 12, 2025, 08:54 PM UTC

   We are currently investigating reports of service disruptions for some users, possible related to specific subscriptions or repos.

2. identified Feb 12, 2025, 09:07 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 12, 2025, 09:08 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Feb 12, 2025, 09:20 PM UTC

   While we have applied a fix and are monitoring for any further issues, we are clearing backlogged jobs for some accounts. If you are waiting on some recent builds to complete, please give them at least another 20 minutes to clear. If you are not seeing your builds clear after that, please reach out to us with your org/subscription and repo name(s) at [email protected].

5. monitoring Feb 12, 2025, 09:43 PM UTC

   We are deploying extra servers to help clear backed up jobs. This will entail a rolling reboot, which may cause some users to lose their current connection to Coveralls.io. Your connection should be restored momentarily, so please try again in 30-sec to 1-min.

6. monitoring Feb 12, 2025, 10:00 PM UTC

   Partial outages. We are working to resolve asap.

7. monitoring Feb 12, 2025, 10:13 PM UTC

   We are still experiencing partial outages as we try to deploy across an extended fleet of servers. We are all hands and working to resolve asap.

8. monitoring Feb 12, 2025, 10:25 PM UTC

   We have resolved the partial outages and are monitoring.

9. monitoring Feb 13, 2025, 05:33 AM UTC

   We are continuing to monitor for any further issues.

10. monitoring Feb 13, 2025, 12:16 PM UTC

    We are not clearing background jobs fast enough to recover by morning US PST, so we will be putting the site into read-only mode for about 2 hrs from (4:30a-6:30a US PST) in order to perform some database operations.

11. monitoring Feb 13, 2025, 01:00 PM UTC

    We are in maintenance mode as we perform some database tasks to improve our performance in clearing background jobs still stuck-in-queue. ETA: 2 hrs. But we may need to update this as we monitor progress.

12. monitoring Feb 13, 2025, 01:33 PM UTC

    Use "fail on error" to keep Coveralls 4xx from failing your CI builds / holding up your PRs: While our API is in maintenance mode, new coverage report uploads (POSTs to /api/v1/jobs) will fail with a 405 or other 4xx error. To keep this from breaking your CI builds and holding up your PRs, allow coveralls steps to "fail on error." If you are using one of our Official Integrations, add: - `fail-on-error: false` if using Coveralls GitHub Action - `fail_on_error: false` if using Coveralls Orb for CircleCI - `--no-fail` flag if using Coveralls Coverage Reporter directly Documentation: - Official Integrations: https://docs.coveralls.io/integrations#official-integrations - Coveralls GitHub Action: https://github.com/marketplace/actions/coveralls-github-action - Coveralls Orb for CircleCI: https://circleci.com/developer/orbs/orb/coveralls/coveralls - Coveralls Coverage Reporter: https://github.com/coverallsapp/coverage-reporter Reach out to [email protected] if you need help.

13. monitoring Feb 13, 2025, 05:15 PM UTC

    Our ETA for reverting maintenance mode is within the next 30 minutes.

14. monitoring Feb 13, 2025, 05:43 PM UTC

    We are out of maintenance mode and monitoring live transactions.

15. resolved Feb 13, 2025, 08:41 PM UTC

    We are closing this issue but will continue to monitor as we clear the remaining queues of background jobs from yesterday. If you believe any of your recent builds are still affected (incomplete), or if you are having any issues uploading coverage reports, please reach out to us at [email protected].

16. postmortem Feb 17, 2025, 05:14 PM UTC

    ### **Incident Postmortem: Database Partitioning Bottleneck & Job Backlog** #### **Summary** A database partitioning limitation caused a severe backlog of background jobs, leading to degraded build processing times from **Monday, February 10, to Friday, February 14, 2025**. The backlog resulted from excessive autovacuum contention on a high-growth table, which ultimately led to cascading failures in job processing, database performance, and monitoring visibility. #### **Root Cause** Several tables in our production database grow at an accelerated rate. While we employ a partitioning strategy to prevent them from becoming unwieldy, our time-based approach failed to transition a critical table before it reached an unmanageable size. During investigation, we found: * The table was in a perpetual state of autovacuum due to an excessive number of dead tuples. * The high volume of tuples prevented autovacuum from progressing beyond the scanning phase, causing tuple locks. * These locks delayed regular transactions, leading to transaction backups that worsened over time. * By late **Tuesday, February 11**, the backlog had reached a breaking point, causing tens—eventually hundreds—of thousands of jobs to accumulate. ### **Impact** 1. **Background Job Delays** 1. A significant job queue buildup occurred between **February 10 and February 11**. 2. Clearing the backlog took an additional two days \(**February 11–13**\). 3. Failed jobs in long-tail retries prolonged the impact for another **24–36 hours**. 2. **Monitoring Gaps & Alert Failures** 1. Average job duration alerts triggered only after the queue size became a critical issue. 2. As server load increased, monitoring metrics stopped logging, preventing alerts that could have provided earlier intervention signals. 3. **Database & Infrastructure Overload** 1. Scaling up resources to clear the backlog introduced additional database contention due to high transaction volumes, exacerbating delays. 2. The increased database load led to degraded server performance, disconnecting our orchestration layer and APM monitoring. 3. This created a self-reinforcing failure loop that required **continuous manual intervention** from **February 12 to February 13**. #### **Resolution** * We transitioned the affected table, significantly relieving the bottleneck. * We scaled up resources to process the backlog, though this required careful throttling to avoid further database contention. * By **Thursday, February 13**, we placed the site into **maintenance mode for 30 minutes** to reduce load—but ultimately needed nearly **two hours** to restore stability. * To prevent immediate re-saturation, we deferred processing some older jobs to lower-traffic periods. * By **Thursday evening**, build times stabilized as overall traffic declined. * By **Friday morning, February 14**, all remaining queued jobs had processed without further intervention. #### **Next Steps** 1. **Finalizing Database Transitions** 1. To fully resolve performance degradation, we transitioned **two additional tables** closely related to the affected table. 2. This was completed during a **maintenance window on Saturday, February 15 \(8 PM – 11:59 PM PST\).** 2. **Long-Term Database Optimizations** 1. We will perform **VACUUM FULL** on legacy tables to remove ~36B dead tuples and optimize disk layout. 2. Further maintenance windows will be scheduled on late-night weekends. 3. **Partitioning Strategy Enhancements** 1. We are evaluating **size-based partitioning** or a refined **time-based strategy with shorter intervals** to prevent similar issues. 4. **Improved Monitoring & Alerting** 1. We will introduce **earlier warning thresholds** to detect job queue buildup before it becomes critical. 2. We will enhance **database contention monitoring** to catch autovacuum failures and lock contention earlier. #### **Conclusion** Even after **12\+ years in production**, incidents like this remind us of the importance of continually evolving our **data management and monitoring practices**. As Coveralls scales, we are committed to refining our approach to proactively address infrastructure challenges before they affect users. We sincerely apologize to all users affected by this incident. If you need assistance with historical builds or workflow adjustments, or if you'd like to share feedback, please contact us at [**[email protected]**](mailto:[email protected]). Your input will help us shape future improvements.

Read the full incident report →

Started Jan 22, 2025, 03:08 PM UTC· resolved Jan 22, 2025, 03:37 PM UTC

1. investigating Jan 22, 2025, 03:08 PM UTC

   We are currently investigating this issue.

2. identified Jan 22, 2025, 03:27 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jan 22, 2025, 03:33 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 22, 2025, 03:37 PM UTC

   This incident has been resolved.

5. postmortem Jan 22, 2025, 06:09 PM UTC

   **Incident**: Service Disruption Due to Failed SSL Certificate Renewal * **Date**: January 22, 2025 * **Duration**: 17 minutes \(07:00-07:17 UTC\) * **Impact**: Service interruption due to SSL certificate issue **Summary**: Coveralls experienced a brief service disruption when our automated SSL certificate renewal process failed. While our SSL certificates auto-renew 30 days before expiration, one unreachable server prevented the renewal process from completing successfully. **Timeline**: * Prior to incident: Multiple automated renewal attempts unsuccessful * 07:00 UTC: Service disruption began * 07:17 UTC: Service restored after infrastructure adjustment **Root Cause**: The incident occurred when one server became unreachable during our SSL certificate auto-renewal process. While our certificates are configured to auto-renew, the renewal process requires successful deployment across our infrastructure. The unreachable server prevented this deployment, ultimately leading to an outage due to “certificate expiration.” **Resolution**: We identified and removed the problematic server from our infrastructure, allowing the SSL certificate renewal and deployment to complete successfully. **Preventive Measures**: 1. Enhanced monitoring for SSL renewal processes 2. Improved early warning system for similar infrastructure issues 3. Updated incident response procedures \(new SOP\) 4. Additional automated health checks We apologize for any disruption this caused and continue working to improve our infrastructure reliability.

Read the full incident report →

Started Dec 09, 2024, 05:29 PM UTC· resolved Dec 09, 2024, 08:45 PM UTC

1. investigating Dec 09, 2024, 05:29 PM UTC

   We are currently investigating report of a partial outage for some users.

2. identified Dec 09, 2024, 06:55 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Dec 09, 2024, 07:55 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 09, 2024, 08:45 PM UTC

   We are considering this resolved after monitoring for ~90 minutes. If you happen to experience any ability to login or access your Coveralls Dashboard, please let us know at [email protected].

Read the full incident report →

Started Oct 31, 2024, 09:08 PM UTC· resolved Nov 03, 2024, 10:47 PM UTC

1. investigating Oct 31, 2024, 09:08 PM UTC

   Several users have reported longer than average build times, either appearing as large number under a build's "BUILD TIME" or as builds staying in PENDING COMPLETION state for longer than expected. We are investigating the issue.

2. identified Nov 01, 2024, 03:30 PM UTC

   The issue has been identified and a fix is being implemented and planned for a no-downtime window this evening. This issue continues to affect some customers and not others, but our fix should address longer build times for affected customers.

3. resolved Nov 03, 2024, 10:47 PM UTC

   We have implemented the fist of several planned changes to hopefully improve builds times for the users who have been experiencing slow build times. We have planned additional changes for one evening this week and the coming weekend. If you're experiencing slower than normal build times, or have any questions, please reach out to us at [email protected].

Read the full incident report →