Coveralls Outage History

Started Sep 30, 2026, 01:19 PM UTC· resolved Sep 30, 2026, 08:11 PM UTC

1. monitoring Sep 30, 2026, 01:19 PM UTC

   User in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST. We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next 45min - 1hr.

2. monitoring Sep 30, 2026, 04:25 PM UTC

   The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here.

3. resolved Sep 30, 2026, 08:11 PM UTC

   This incident has been resolved.

4. postmortem Oct 11, 2026, 06:55 PM UTC

   # Update: Major Performance Optimization Released for Heavy Repo Builds Triggered by this latest incident, we've released a major optimization to our code that processes large \(aka. “heavy”\) repos \(>=5K source files\). Tests over the weekend of Fri, Oct 9-Sun, Oct 11 show a near 20x improvement in performance per heavy repos build at lower, weekend volumes. While we expect the optimization to continue to drastically improve build times for these repos \(and reduce instances of increased latency for them\), we won’t know what the aggregate improvement is until we are at full volume this week. We are monitoring to get a sense of where the performance improvement lands, but are hopeful we’ve crossed a significant milestone in how efficiently these builds are processed given the unique challenges many of them pose.

Read the full incident report →

Started Sep 28, 2026, 04:03 PM UTC· resolved Sep 28, 2026, 05:21 PM UTC

1. investigating Sep 28, 2026, 04:03 PM UTC

   We are currently investigating this issue.

2. identified Sep 28, 2026, 04:11 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 28, 2026, 04:13 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Sep 28, 2026, 04:14 PM UTC

   Operational in all regions. We are continuing to monitor for any further issues.

5. monitoring Sep 28, 2026, 05:02 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Sep 28, 2026, 05:21 PM UTC

   This incident has been resolved.

7. postmortem Sep 28, 2026, 06:47 PM UTC

   ### Resolved: elevated error rates, Monday September 28, 08:00 to 09:15 PT **Timeline**: Between about 08:00 and 09:12 PST on September 28, some requests to Coveralls returned HTTP 500 errors. Coverage uploads, GitHub webhook events, badges, and coverage pages were affected. Failures were intermittent at first and peaked toward the end of the window. Service was fully restored at 09:12 PST. **Cause**: a capacity issue with our primary database. We have have added capacity, enabled automatic scaling, and changed our monitoring triggers to be alerted well before this kind of issue can recur. **What to do**: If you were affected: re-run any CI job that uploaded coverage to Coveralls between 08:00 and 09:15 PT on September 28, or simply push a new commit. No data was lost or corrupted; those uploads simply did not land. We are sorry for the disruption.

Read the full incident report →

Started Aug 11, 2026, 07:02 PM UTC· resolved Aug 11, 2026, 09:09 PM UTC

1. investigating Aug 11, 2026, 07:02 PM UTC

   We are currently experiencing occasional "Website under heavy load" warnings. We believe this is associated with an army of scraper bots we experienced and successfully mitigated this weekend. We believe those bots may have gotten around an initial phase of defense involving managed challenges at our edge provider. Separate from a DDoS attack, these are millions of individual requests coming from over 200K different IP addresses across 10 different countries, so our standard DDoS defense of IP quarantines at our firewall do not apply. We are implementing a second phase of defense and will report updates here.

2. monitoring Aug 11, 2026, 07:55 PM UTC

   This was the second wave of the attack we experienced this weekend, broadened and tripled in volume. We implemented a second phase of defense which appear to be working and will continue monitoring for the next hour.

3. resolved Aug 11, 2026, 09:09 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 15, 2026, 05:28 PM UTC· resolved Jul 20, 2026, 05:32 PM UTC

1. monitoring Jul 15, 2026, 05:28 PM UTC

   All systems fully functional. We have identified increased latency for large repos again and have proactively arrested sources of excess requests. We expect to restore normal latency within the hour.

2. monitoring Jul 15, 2026, 06:14 PM UTC

   We have reduced traffic in affected queues by 75%. We still expect to clear the queue within about 1 hr. We will post updates here. If you think you may have been paused as a user with a large repo (>=5K source files) that has uploaded 3-10x normal daily uploads, please reach out to us at [email protected] and we'll confirm and give you some next steps to avoid further pauses (see Usage Add-Ons at https://coveralls.io/pricing).

3. monitoring Jul 16, 2026, 12:47 AM UTC

   We are continuing to monitor for any further issues.

4. monitoring Jul 16, 2026, 06:25 PM UTC

   We are continuing to proactively monitor and clear the large repo queues. We are working on a longer term solution over the weekend and will post about it in the postmortem for this issue when complete. We will keep this incident open as long as we have large repo queues with above-average latency.

5. monitoring Jul 17, 2026, 03:48 PM UTC

   Latency remains elevated by about 30% for large repos. We are continuing to monitor and do what we can to mitigate until we can implement a longer-term solution.

6. resolved Jul 20, 2026, 05:32 PM UTC

   This issue was resolved over the weekend. We will continue to monitor for elevated latency in the modified queues.

Read the full incident report →

Started May 18, 2026, 04:41 PM UTC· resolved May 28, 2026, 03:53 PM UTC

1. identified May 18, 2026, 04:41 PM UTC

   We have received reports of increased report latency for larger projects (5K file and up). We are doing our best to clear the background job queues for these projects. If you have one of these projects, feel free to reach out and we'll check status for you.

2. monitoring May 18, 2026, 05:37 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring May 19, 2026, 01:36 PM UTC

   We are still receiving reports of latency for large projects and are seeing backlogs of background jobs in queues dedicated to large repos (5K+ source files). We may pause some outlier repos with excessive numbers of jobs in order to clear general traffic. If you believe you may have been paused, reach out and we'll let you know.

4. monitoring May 20, 2026, 01:48 PM UTC

   We are keeping this incident open as we continue monitoring.

5. monitoring May 20, 2026, 02:06 PM UTC

   We have moved all repos with more than 300 jobs in queue to dedicated queues for processing. These 7 repos were responsible for 75% traffic in our heavy repos queue. This move allows us to provide normal processing times for non-outlier repos and dedicated resources to outlier repos. Outlier repos by their job volume will still take longer to clear. If you believe you may be one of these outlier repos, please reach out and we'll confirm and give you an idea of when your jobs will be processed. If you'd like faster processing, we can also pause processing on older job, for instance, anything older than 1 hour (or 2 hours, or 3 hours) ago.

6. monitoring May 21, 2026, 04:23 PM UTC

   We continue to monitor background processing queues for larger repos (5K+ files), as we continue offloading incoming jobs from the outlier repos whose volume led to the original backlog. If you think you may have been paused as one of these repos with outlier-level volume, please reach out to us at [email protected] and we'll confirm status and get you an ETA on when your remaining jobs will be processed.

7. monitoring May 25, 2026, 08:06 PM UTC

   We are continuing to monitor for any further issues.

8. monitoring May 27, 2026, 10:18 PM UTC

   We've cleared the backlog of background jobs for larger projects from our heavy project queues, with the exception of nine projects whose volume continues to exceed our daily fair use thresholds. Those nine have been redirected to secondary and tertiary queues that drain only after the primary queue is clear. Most of their jobs have now been processed, but a few projects generate more jobs than a normal day + overnight cycle can drain, so some backlog remains. We're continuing to work those queues down manually. If you think one of your projects may be affected, reach out to us at [email protected] and we'll check status for you.

9. resolved May 28, 2026, 03:53 PM UTC

   We are closing this incident. Our main background processing queue for larger repos has had no backups in 48 hrs. We will continue monitoring for recurrence.

Read the full incident report →

Started May 10, 2026, 02:44 PM UTC· resolved May 10, 2026, 03:11 PM UTC

1. investigating May 10, 2026, 02:44 PM UTC

   We are currently investigating an issue with an apparently expired SSL cert. Our orchestration layer shows no issue with certs so we need to dig deeper.

2. resolved May 10, 2026, 03:11 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 24, 2026, 10:52 PM UTC· resolved Feb 27, 2026, 10:30 PM UTC

1. investigating Feb 24, 2026, 10:52 PM UTC

   We are currently investigating this issue.

2. investigating Feb 24, 2026, 10:58 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 25, 2026, 01:36 AM UTC

   The issue has been identified and a fix is being implemented.

4. identified Feb 25, 2026, 03:28 AM UTC

   Outage will be extended for at least several more hours. To avoid further disruption to your CI workflows, we recommend employing the fail-on-error: false input option available with all official coveralls integrations. See: https://docs.coveralls.io/integrations#official-integrations. Official Integration: fail-on-error option - GitHub Action: `fail-on-error: false` - CircleCI Orb: `fail_on_error: false` - Coverage Reporter (CLI): `--no-fail`

5. monitoring Feb 25, 2026, 05:48 PM UTC

   We are experiencing an outage that appears to have originated at our infrastructure provider. We are unable to access our account and are working with our account manager to resolve the issue. Unfortunately, due to the nature of the incident, we have no way to establish an ETA. We are doing everything we can to restore service, and will update our status page as soon as we have a fix or an ETA. Please follow along here: https://status.coveralls.io/ You can subscribe for email updates by clicking SUBSCRIBE TO UPDATES button in the top right corner of this page.

6. monitoring Feb 25, 2026, 05:49 PM UTC

   We are continuing to communicate with our hosting provider around any updates or ETA.

7. monitoring Feb 26, 2026, 01:56 AM UTC

   After a full day of working with our hosting provider's account team (who've been really helpful), we've discovered that this outage stems from an account restriction that, by all indications, shouldn't have happened (we believe the technical term is "snafu"). They are actively working to get it reversed. This is not a technical failure — our infrastructure and all customer data are fully intact. Services will be restored as soon as account access is. We're hopeful for resolution this evening or tomorrow morning. We'll have more to say once we're back online.

8. monitoring Feb 26, 2026, 02:53 PM UTC

   As of 6am PST Thu, Feb 26, we are still waiting on restoration. We thought this would be resolved by this morning. We are in touch with our account team and are doing what we can to restore service. We apologize for the ongoing inconvenience.

9. monitoring Feb 27, 2026, 01:13 AM UTC

   Day 3 | Evening Status: Unresolved — No ETA We want to be transparent with you: after two full days of working directly with our hosting provider’s account team, we still do not have an ETA for service restoration. Despite having internal sponsorship within our provider’s organization, we have been unable to get the traction needed to resolve what appears to have been an automated action, one that no one on their side has taken ownership of causing or fixing. We understand the severity of this disruption. Your CI workflows depend on Coveralls, and we can see the impact this outage is having on your teams. That is not lost on us for a single moment. This experience has made something painfully clear: no matter how trusted or established a cloud provider may be, a single vendor dependency without a failover plan is an unacceptable risk, for us and for you. We are already evaluating backup infrastructure options to ensure that nothing like this can take our service offline this way again. We’ll share more in our full postmortem. In the meantime, to those of you who have stayed patient and stuck with us through this, thank you. We see you, we appreciate you, and we are committed to making this right. If there is anything we can do to help in the interim, please reach out. We will continue to post updates as we have them.

10. monitoring Feb 27, 2026, 03:41 PM UTC

    Day 4 | Morning Status: Unresolved — No ETA It's the morning of Day 3, and service has still not been restored. We want you to know we're here, fully aware, and have not stopped working on this. We remain in constant communication with our hosting provider's account team, who we have every reason to believe is continuing to advocate for us internally. But the reality remains the same: we do not have an ETA, and we do not have direct control over the outcome we need: restoration of our services and access to our infrastructure. As we pass milestones in this outage that we never imagined we would see (and haven't in 15 years of operation) the path forward is becoming clearer. We need to find a new provider, at minimum for failover, and potentially as our primary host. We've been reading the tweets, watching the YouTubes, and evaluating options. But if anyone in a position similar to ours has migrated away from a primary tier host and successfully manages a write-intensive service with hundreds of thousands of daily uploads, we would genuinely love to hear from you about your experience. Please reach out: [email protected] Because if our services are not restored by the end of today, we will be standing up new infrastructure over the weekend. We will continue to post updates as we have them.

11. monitoring Feb 27, 2026, 07:02 PM UTC

    We are seeing systems coming back online. We assume service has been restored at some level. We will assess and update you here ASAP.

12. monitoring Feb 27, 2026, 07:11 PM UTC

    We are back, for at least some users. We’ll stay at “partial outage” until we confirm full restoration.

13. monitoring Feb 27, 2026, 07:19 PM UTC

    We would love to hear from anyone outside of the US as to whether service has been restored for you: [email protected] We have no reason to believe service hasn't been restored to all regions, but we want to make sure. We will continue monitoring.

14. monitoring Feb 27, 2026, 08:23 PM UTC

    After 1-hour of monitoring, we are cautiously optimistic that service is fully restored in all regions. Feel free to confirm service in your area (with thanks): [email protected] And please let us know if you are still having any issues: [email protected] We will continue to monitor throughout the rest of the day, and weekend. Finally, please come back for our post-mortem. We are aiming for the first part of next week, but our priority is making sure service is stable. For all of the kindness and support, we thank you. To everyone, thanks for your patience. We'll be in touch.

15. resolved Mar 02, 2026, 06:10 PM UTC

    This incident has been resolved.

16. postmortem Mar 09, 2026, 05:37 PM UTC

    # Suspended for Not Paying—While Paying _Latest update: Tuesday, Mar 24_ [This post-mortem is also available as a PDF](https://s3.amazonaws.com/assets.coveralls.io/statuspage/incidents/20260224/postmortem/Coveralls-Post-Mortem-Service-Outage-Feb-2026.pdf). ## Overview On February 24, our hosting provider suspended our account. Our service was down for 68 hours. The stated reason was past-due charges. When we reached our account manager on the day of suspension, he reviewed our account and told us what he saw: an "account restriction" that, in his words, "shouldn't have happened." Bizarrely, his review turned up an internal note claiming we'd made _no payments in six months_. That was demonstrably false and we provided immediate evidence to prove it. We did have an outstanding balance—the product of a billing dispute that took nearly a full year to reach a decision in our favor. But finalizing what we owed required tracking down two credits we couldn't locate. We asked AR one specific question: _where did those two credits go?_ We asked it _six times_. For some reason we still don't understand, they simply stopped responding. We escalated to our account manager, who promised to intervene. Then we heard nothing from anyone. Until we were suspended. Despite providing evidence of our payments, open billing cases, and relentless communication, we waited three days for our service to be restored. When someone finally acted, we were back online within minutes. Three days, and then minutes. Our provider has not explained that gap, or responded to the formal request for answers we submitted more than two weeks ago. Without those answers, what follows is our account. We're sharing everything and letting you judge for yourself. _If what you need most right now is assurance that this won't happen again, and to know what we're doing for affected customers, you can jump ahead to **What We're Doing** and **Making It Right**._ But we'd ask you to read through. On the surface, this looks like a company that didn't pay its bills and got suspended. The full picture tells a different story, and the evidence speaks for itself. ## About Coveralls Coveralls has been providing code coverage analytics for 15 years. We're bootstrapped: no outside funding. Over 225,000 developers use [Coveralls.io](http://Coveralls.io) every day, and roughly 90% of them are open-source contributors who use the service for free. \[1\] We have never experienced an outage like this in our 15-year history. ## The Incident ### What We Were Told About Suspension More than once, over the past year, we asked our account manager directly whether we were at risk of suspension. Each time, the answer was no. He told us suspension was designed to be rare, that it was reserved for accounts that had gone _delinquent_—stopped _paying_, and stopped _communicating_—and that it required his sign-off, _and_ his manager's. He told us he received a regular list of at-risk accounts among those he managed, so he could review them and give feedback before any action was taken. He told us we had _never_ appeared on that list. He said our provider _knew_ we were paying, and _knew_ we were actively communicating, with AR and with him. Based on everything he told us, suspension wasn't just unlikely, it wasn't supposed to be possible. At least not without warning. ### Timeline **Tuesday, Feb 24 — Day 1** 1:45 PM PST: We suddenly lose access to our hosting provider's console. The service is still up, so we assume it's a technical issue. Multiple team members try their credentials. None work. 2:42 PM: We receive our first support request saying, "posting coverage data stopped working", but we don't see it until 2:51 PM due to the next event. 2:45 PM: A team member gets past a login screen and sees a message: "Your account was suspended because of past due charges." We don't believe it. We reach out to our account manager by email, then text, then phone. He confirms an "account restriction" that, based on his review of our account, "shouldn't have happened", and shares that an internal note on our account says, "_Customer has made no payments for 6 months_." 2:51 PM: We see the first support request. 2:52 PM: We open an incident on our status page. At this point, we are still expecting this to be reversed quickly and do not yet disclose the suspension publicly. In hindsight, we should have. ~3:00 PM: While we're on the phone with our account manager, one of us discovers the official suspension notification in our inbox, timestamped 2:03 PM, about 15 minutes after we were first locked out. Our AM asks us to create a new support ticket and promises to escalate. Two hours fly by while we feverishly collect and send evidence of improper suspension. We expect our AM to get this reversed as soon as he gets through to—_someone_. 5:36 PM: No definitive update. Our AM is "still working on it." He tells us the AR team involved is on Eastern time and may be gone for the day. We post that the issue has been identified and a fix is being implemented. We believe this. We are wrong. 7:28 PM: No resolution. Our account manager tells us he continues to escalate but that under the current circumstances, he himself has "limited access" to our account. Still—_should_ be fixed by morning. We update our incident, extend the outage window, and recommend fail-on-error: false for CI workflows. ~9:30 PM: Still no resolution. We check status throughout the night. **Wednesday, Feb 25 — Day 2** 6:45 AM: We text our account manager, surprised service has not been restored. 9:48 AM: First public disclosure — outage originated at our infrastructure provider. We are unable to access our account. 5:56 PM: After a full day working with our provider's account team, we learn more about the restriction. Not a technical failure—infrastructure and data intact. Their team indicates they are working to reverse it, and we expect resolution by morning. **Thursday, Feb 26 — Day 3** 6:53 AM: Still waiting. "We thought this would be resolved by this morning." 5:13 PM: After a full second day of trying to get answers as to why this is happening, and why it hasn't been resolved yet, the picture remains unclear. No ETA. "What appears to have been an automated action, one that no one on their side has taken ownership of causing or fixing." We commit to evaluating backup infrastructure and promise a full post-mortem. **Friday, Feb 27 — Day 4** 7:41 AM: "As we pass milestones in this outage that we never imagined we would see \(and haven't in 15 years of operation\)..." We announce that if not restored by EOD, we will stand up new infrastructure over the weekend. 11:02 AM: Systems begin coming back online. We have not been told what changed. 11:11 AM: Service partially restored. 12:23 PM: After one hour of monitoring, cautiously optimistic that service is fully restored in all regions. **2:30 PM: After two more hours of monitoring, the incident is officially marked resolved. Total outage: approximately three days \(68 hours\) across 4 calendar days.** ‌ ## What We Think Happened Here's what we found when we started looking for answers: ### The Note That Wasn't True The first thing we learned about why our account had been suspended was what our account manager told us: there was an internal note from AR that read: > _Customer has made no payments for 6 months._ This was demonstrably false. Here is our payment history for that exact period—September 1, 2025 through February 24, 2026 \(the date of our suspension\):  We shared this evidence with our provider immediately, backed by bank statements:  ‌ But it raises the question: if we were paying, why would our provider think we _weren't_? ### Why The Note Existed When we look at our billing console we can see why someone might _think_ we hadn't paid: **it shows none of our payments**. Invoices we know to have been fully paid show none of our payments against them. Most notably, the **Transactions Tab** of the **Payments** section of our billing console contains _**no payment records**_ **for last 6 months**—resonating with that internal note: ‌ ‌  ‌ If an automated system relied on this same data to determine whether a customer had made payments, and that data showed six months of silence, it _might_ trigger exactly the kind of suspension we experienced: an automated action with no advance warning, with no one on their side able to explain it or willing to take ownership of it. This is our strongest piece of evidence that the suspension was triggered by bad data, and not our actual account status. — To put this in context: over the six months during which the internal note claimed we'd paid _nothing_, our provider billed us $127,186. Our verified payments for the same period totaled $120,312—a gap of roughly $6,900, which is less than a third of the amount we were actively disputing in our second open billing case.   \(By March 5, our payments even exceeded our total invoiced charges for the period: we were not delinquent.\) Our payments were in line with our usage. We weren't _up-to-date_—but the portion of the balance we still carried traced directly back to a single incident that triggered a billing dispute, which took ten months to resolve and still hadn't fully landed by the time we were suspended. — But the note, and the billing console, only explain so much. They might explain how the suspension was _triggered_. They don't explain why it took _three days_ to reverse, even after providing evidence of our payments and open billing cases. If there's an answer for that, it lives in the rest of the story. We've looked. We don't see a justification for either. Draw your own conclusions. ### How The Balance Happened In 2025, we faced a critical infrastructure challenge. Our production PostgreSQL database, approaching a 65TB ceiling due to failing maintenance operations, also needed a major version upgrade before end-of-life. We started the upgrade a month early. It took four-and-a-half months to complete. During three-and-a-half of those months, our provider levied Extended Support Fees on us that roughly _quadrupled_ our database hosting costs for that period. \[2\] We asked our provider to pause or waive them—the fees were simply beyond what we could absorb. They told us that wasn't possible, but that we could open a billing dispute and make our case. We did that, and our provider eventually approved credits for the full amount. But that process took ten months. \[3\] During those ten months, those fees accumulated into a past due balance, so we asked AR for a repayment plan, and their answer was: wait. The credit outcome would affect what we owed, and they needed the case to resolve before we could finalize any repayment agreement. In response to an email titled "Seeking repayment plan for past due balance," an AR representative told us: > _Upon checking, the support case related to billing adjustment review via case ID remains active. Please continue to monitor the support case for any update and provide the required information for faster resolution. Once the billing adjustment review for re-appeal was completed, we can discuss for possible payment plan._ In the meantime, we paid what we could and waited for resolution. Our balance stayed on the books for ten months—and with it came past-due notices with threats of suspension we were told were automated and couldn't be stopped. Those are what prompted us to ask our account manager, more than once, whether we were truly at risk of suspension, and his answers gave us confidence that, difficult as the situation was, we were managing it the right way. When those warnings arrived, we did the same thing: made a payment and notified AR. That pattern—always paying, always communicating—put us squarely in the playbook our account manager described, and it seemed to work because we never appeared on his at-risk list. ### A Balance We Couldn't Pin Down When the billing case finally resolved in January 2026, we expected to finalize a repayment plan, but we were blocked _again_—and explaining why requires a brief note on how the approved credits worked: The credits didn't arrive as a direct reduction to our outstanding balance. They came in eight different amounts, apparently as cash refunds—separate transactions that made their way back to us through different payment methods. This meant our balance didn't shrink by the amount of the credits as we expected it would; and, more importantly, some of the credits remained outstanding due to two transactions we couldn't locate. Whether and how _those_ had been applied—_unclear_. Those two missing refunds had no payment method attached in our billing console, and we couldn't find matching deposits in any of our bank or card accounts. They were material to calculating our actual outstanding balance, and any repayment plan, and without AR's help, we couldn't track them down. So we submitted our first request for help on February 12: > _We are having trouble balancing our remaining amount due \($\[redacted\]\) against our credits awarded Jan8 \($\[redacted\]\). Can you help us track down these two \(2\) refund transactions? \[...\] We can't find a payment method associated with those two refunds \[...\] And we can't find matching transactions anywhere in any of our card or bank account statements. Do you know how / where those refunds were applied?_ Three days later, on Feb 15, we received a response that simply restated the information in our billing console. It didn't answer our question. Within an hour of receiving it, we responded and made the stakes explicit: > _Sorry if I wasn't clear: We have not been able to find these two refunds: \[...\] We cannot find the money. We believe we did not actually receive it. We have a general repayment plan in mind that we hope to present, but first we need to resolve this question of these two missing refunds. Without being able to account for that $\[redacted\], we believe our current outstanding balance could be off by that much. Your input on this is crucial for us._ We received no response. Growing concerned about a possible disconnect with AR, that Friday, February 20, we cc'd our account manager on another follow-up to AR and, separately, forwarded the full thread to him directly, asking him to intervene: > _Is there any way you can help us get a response from \[Redacted\] or someone else in Billing regarding the missing refund payments? We want to implement a repayment plan, but need to be clear on our numbers._ He replied the same day: > _Yes — I'll reach out to her. \[...\] Let me pop into the ticket to see if I can escalate._ That was the last we heard from anyone. On February 19, we received a past-due notice with a suspension warning dated the following day. This wasn't unusual—we'd received them throughout the entire ten-month period and had always responded the same way: we made a payment and notified AR. We made a $10,000 payment on February 20 and let AR know. We interpreted the passing of February 20 the way we always had: as confirmation that our payment had been received and the threat had passed. Four days later, we were suspended. ### Six Weeks Before Suspension Six weeks before our suspension, on January 14, we filed a second billing dispute. We had been trying to _reduce_ our hosting costs by shrinking our production database; instead, a mandatory storage configuration upgrade blocked us for 27 days and roughly _doubled_ our costs during that period. We disputed the incremental charges. After working through all required questions, our request was formally submitted to Billing's specialist team: > _Since these charges are not yet present on your account we here at Billing team cannot promise a refund or credits to offset these charges. However, we can connect with your AWS Account Manager for getting an exception approved for you by our specialist internal team. Please contact your Account Manager so they can work with the team for the required approval._ We formally looped in our AM as directed—he was already cc'd on the case, but we followed the process—and he confirmed he was actively monitoring and advocating for our credit request. On January 22, we experienced a temporary account restriction—we were unable to add certain services needed for our workaround. We reached out to our AM and sent a parallel update to our AR contact, informing her of the open billing case and our AM's involvement. The restriction was lifted. Our AR contact replied with direction nearly identical to what we'd received throughout our first billing dispute: > _Kindly continue to monitor the support case for billing adjustment review._ The technical workaround we selected took 20 days—January 22 through February 10—spanning our two most recent billing periods. When the effort was complete, we updated Billing, and our account manager, with the final timeline, cost, and the amount of our credit request. By February 12, everyone had the same information: the work was done, the dispute was submitted, and our last two invoices were under active dispute. Based on prior experience and similar direction from AR, we believed charges under active dispute would not be counted toward our outstanding balance—let alone any balance used to justify suspension. We don't know whether our disputed billing charges _were_ part of the outstanding balance that triggered suspension, or not. All we know is that AR knew about this open case, and so did our AM, who had told us more than once that he reviewed at-risk accounts before any suspension action was taken. Neither warned us that suspension was imminent. — To recap: we were paying—over $100K during the same six months the internal note claimed we'd paid nothing. We were communicating—multiple messages to AR in the twelve days before suspension, plus a direct escalation to our account manager after AR went silent. We had an open billing dispute, and a repayment plan in progress with a specific blocker we had been asking AR to help us resolve for nearly two weeks. None of the criteria our account manager described for suspension applied to us. We hadn't stopped paying. We hadn't stopped communicating. We had never appeared on his at-risk list. The suspension hadn't had his sign-off or his manager's. On the day of our suspension, our account manager reviewed our account and told us it "shouldn't have happened." Three days later, when someone finally acted, service was restored within minutes. To this day, we still don't know exactly why the suspension happened, or why it took three days to reverse. ## Open Questions \(Our Formal Request for Answers\) On March 4, three business days after our service was restored, we sent a formal request for answers to our hosting provider. As of this writing—fourteen days later—we have not received a response. We're not publishing those questions verbatim here because we don't want to frame them as a public ultimatum. But our questions cover four areas: We want to understand what triggered the suspension and who authorized it—whether it was automated or a human decision, and why we were suspended when we had never appeared on our account manager's at-risk list. We want to know why our billing console showed no payment records for a period in which we had paid over $104K in verified bank transactions, and what prompted the internal note. We want to know why our missing refund issue—raised six times between February 12 and February 24—was never substantively addressed before we were suspended. And we want to understand why reinstatement took three days after we provided evidence of our payments, our open billing cases, and our multiple attempts to resolve the blocker to our repayment agreement—especially when service was restored in _minutes_ once someone finally acted. We're committed to updating this post-mortem when we have those answers. ## What We've Learned Two things we should have done differently, and one assumption we should never have made. The process guidance our account manager provided was verbal. We relied on it in good faith—it was specific, consistent, and came from someone with direct knowledge of our account. But we never sought written confirmation. We should have. We noticed discrepancies in our billing console earlier in this process. We flagged them but didn't fully understand their significance, or their potential connection to how our account might be evaluated—until it was too late to escalate effectively. That's a failure we must own. And finally: we assumed we were safe. We were doing everything we'd been told to do. The most recent suspension warning had passed without incident—as others had. We interpreted that silence as confirmation. It wasn't. We should have insisted on written confirmation of our account status; and when we couldn't get AR to respond, we should have escalated beyond email and beyond our account manager. — That's our account of what happened. What follows is what we're doing about it. ## What We're Doing The most significant thing we've done to prevent a recurrence is reduce our monthly database hosting costs by approximately 50% from their peak. We accomplished this by completing a major database project we'd been working toward for over a year—the same project that was blocked twice, producing the unplanned cost spikes at the center of both billing disputes. That pressure is gone. We can easily meet our regular charges going forward. We have nearly paid off our outstanding balance—by 50% if disputed charges are included, by more if not—and we've already begun implementing our repayment plan. We met with our provider this week to formalize that, but we're not waiting to execute. If we prevail in our open billing dispute, the remaining balance shrinks further, making repayment that much easier and faster. We are standing up failover infrastructure on a second provider, expected to be operational within 60 days. No single vendor should ever again have the ability to take our service offline with no recourse. ### Going Forward Going forward, we will not rely on verbal guidance or implicit signals to determine our account standing. We will seek explicit, written confirmation of our status from both our account manager and a dedicated AR contact—someone we hope will commit, in writing, to direct communication \(not system warnings\) that gives us at least one opportunity to resolve any issue before adverse action is taken. We will also get in writing the exact steps our provider follows before suspending an account. Finally, we owe you a commitment about how we communicate during incidents. During this outage, there were moments—particularly on Day 1—where we knew more than we shared publicly. We held back because we didn't fully believe what was happening, and because we kept expecting it to be reversed at any moment. That was the wrong call. Going forward, we will share what we know as soon as we know it, even when it's embarrassing, scary, and we don't yet have the full picture. You deserve that transparency in real time, not just in a post-mortem. ## Making it Right Every paying customer affected by this outage will receive a credit equal to four days of their current subscription—one day for each day of interrupted service. This credit will be applied automatically to your next bill. We understand that's not enough—it's an attempt to be equitable while not harming ourselves more than we've already been harmed by this outage. We want you to know that if anyone feels it doesn't properly compensate for how the outage affected you, just reach out and we will refund the whole month. No questions asked. To our open-source users: the best we can offer is this post-mortem. We know some of you have been through similar things. If it's helpful for you to discuss further, please reach out. We'd also be grateful to hear from anyone who's been through something similar and what you did about it from a hosting perspective. And thanks for the HugOps. We're ready to share some back now. If you have questions about anything in this post-mortem, we're at [[email protected]](mailto:[email protected]). ## Acknowledgments Finally: thank you. During the worst week in our 15-year history, our community showed up. Customers offered server rack space. An SRE director offered to apply pressure through his own provider relationships. People we'd never met sent messages of support and solidarity. In the DevOps community, the term is: HugOps. We received a lot of it that week, and it made a real difference. For those who've stayed with us through this—thank you. We know you have other options. We appreciate the opportunity to make this right. — ## Notes \[1\] _That ratio matters for this case. Our revenue doesn't scale proportionately to our infrastructure costs. Unexpected cost spikes hit us harder than they would a company whose hosting bill is covered by proportional revenue, which is part of why the billing situation at the center of this story developed the way it did._ \[2\] _AWS RDS Extended Support Fees \(ESF\) are charged per vCPU per hour on any RDS instance running a PostgreSQL major version past its end of standard support date — on top of normal instance costs. Our production instance was a db.r6g.16xlarge: 64 vCPUs, 512 GB RAM, at the 65TB maximum RDS storage allocation. At 64 vCPUs and the Year 1 ESF rate, the surcharge approached the cost of the instance itself. Our Major Version Upgrade — which took four and a half months, three and a half of which fell within the Extended Support window — required a Blue/Green Deployment, meaning we ran two db.r6g.16xlarge instances simultaneously for that entire period, each accumulating ESF charges around the clock. Two instances at normal cost plus ESF on both: that is what nearly quadrupled our database hosting costs for the period._ \[3\] _Note: In connection with the credit award, we signed an agreement that restricts us from disclosing specific amounts or characterizing our provider's position on the dispute._ — [Download a PDF version of this post-mortem](https://s3.amazonaws.com/assets.coveralls.io/statuspage/incidents/20260224/postmortem/Coveralls-Post-Mortem-Service-Outage-Feb-2026.pdf).

Read the full incident report →

Started Feb 02, 2026, 05:33 PM UTC· resolved Feb 02, 2026, 05:33 PM UTC

1. resolved Feb 02, 2026, 05:33 PM UTC

   Just a note to address the gap in our Coverage Calculation Background Job Dequeue Time Graph today, MON, FEB 2 from 00:15:00 PST to 07:35:00 PST: Coveralls experienced no disruption in service at this time. Instead, a deployment issue cause the cron job that reports the metric to fail until it was resolved at 07:35:00 PST.

Read the full incident report →

Started Dec 19, 2025, 02:34 PM UTC· resolved Dec 19, 2025, 03:24 PM UTC

1. identified Dec 19, 2025, 02:34 PM UTC

   We are continuing to experience elevated latency in APAC and EU.

2. monitoring Dec 19, 2025, 02:52 PM UTC

   We are continuing to monitor as we scale to clear backlogged jobs.

3. resolved Dec 19, 2025, 03:24 PM UTC

   We had to pause some queues to recover performance for new jobs, but will clear those as soon as we've recovered normal build times for new jobs (ETA: 15-min). If you are a user in APAC or EU, you may have had your jobs paused. One repo in particular has represented 90% overnight workload. We will reach out to that user.

Read the full incident report →

Started Nov 17, 2025, 04:35 PM UTC· resolved Nov 17, 2025, 07:43 PM UTC

1. monitoring Nov 17, 2025, 04:35 PM UTC

   We received reports from EU customers of elevated latency and have resolved the cause. We will be managing workloads to ensure standard processing times for all new builds, and scaling resources to clear delayed builds. Follow this incident for updates.

2. monitoring Nov 17, 2025, 05:28 PM UTC

   Build times are returning to normal for all builds created since 05:00AM PDT. Older builds are being processed by scaled resources and should clear in 45min to 1 hr. If you are having issues with a slow or stuck build, especially if priority, feel free to reach out to us at [email protected]. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number.

3. resolved Nov 17, 2025, 07:43 PM UTC

   This incident has been resolved. Build times for all new builds is normal across the board. We are still clearing a backlog of background jobs, which should be clear in the next 15-20-min. If you are having issues with a slow or stuck build, feel free to reach out to us at [email protected]. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number.

Read the full incident report →

Started Oct 24, 2025, 06:35 PM UTC· resolved Oct 24, 2025, 09:54 PM UTC

1. monitoring Oct 24, 2025, 06:35 PM UTC

   We received reports of 504 Gateway Timeouts from our Web Servers (coverage report uploads) between 6:00am-6:20am PDT and several moments ago between 11:15am-11:30am PDT. We have recovered service to all web servers and are monitoring for further occurrences.

2. resolved Oct 24, 2025, 09:54 PM UTC

   We are closing this incident having received no further reports of 504 errors today. We will continue to monitor for them.

Read the full incident report →

Started Oct 13, 2025, 06:02 PM UTC· resolved Oct 13, 2025, 11:15 PM UTC

1. monitoring Oct 13, 2025, 06:02 PM UTC

   We have received some reports of 504 errors this morning. We are monitoring to avoid further occurrences. We are looking into any gaps in our new monitoring regime, which has drastically reduced occurrences, to see where we can further improve.

2. resolved Oct 13, 2025, 11:15 PM UTC

   We have received no further reports of 504 timeout errors on coverage report uploads today, but we continue to monitor and will continue trying to improve our mitigations.

Read the full incident report →

Started Oct 10, 2025, 03:23 PM UTC· resolved Oct 10, 2025, 07:00 PM UTC

1. monitoring Oct 10, 2025, 03:23 PM UTC

   We've received results of 504 Timeout errors from 7am PDT through 8:15am PDT. We have addressed the issue and issue was resolved as 8:19am PDT. We are monitoring for further issues.

2. monitoring Oct 10, 2025, 04:46 PM UTC

   Based on the details of today's incident, we've added another layer of monitoring that should give us earlier indications of this error state and allow us to respond more quickly.

3. resolved Oct 10, 2025, 07:00 PM UTC

   This incident is resolved. We have applied an additional layer of monitoring that should help us catch these cases earlier.

Read the full incident report →

Started Sep 26, 2025, 03:28 PM UTC· resolved Sep 29, 2025, 03:56 PM UTC

1. monitoring Sep 26, 2025, 03:28 PM UTC

   All systems operational. During our US overnight hours (PDT / GMT-7), we received additional reports of 504 timeout errors affecting coverage uploads. The issue has since been resolved, and we are actively monitoring for further occurrences. We have already implemented several mitigations over the past few weeks, and additional measures will be deployed today to further reduce the likelihood of these errors, particularly for our international customers. If you experience a 504 error, the most helpful details you can provide are: 1. A timestamp or timeframe of the error 2. Any Cloudflare Ray ID shown on the error page Please share these details in our public tracking issue here: https://github.com/lemurheavy/coveralls-public/issues/1824 These reports are invaluable as we continue to investigate and refine our mitigations.

2. resolved Sep 29, 2025, 03:56 PM UTC

   We are closing this incident after recent mitigations and a weekend without any reports. We continue to implement mitigations and infrastructure changes we believe will further reduce incidents of this error type.

Read the full incident report →

Started Sep 03, 2025, 06:00 PM UTC· resolved Sep 25, 2025, 12:53 AM UTC

1. monitoring Sep 03, 2025, 06:00 PM UTC

   We’re currently seeing elevated reports of 504 Timeout errors affecting some customers on a subset of Coveralls pages, including: - Source File pages - Repo pages - Add Repos pages All systems and pages are generally operational; a subset of customers are experiencing these errors, sometimes intermittently. There is a public tracking issue for the Source File timeout errors here: https://github.com/lemurheavy/coveralls-public/issues/1757 Fix in progress: We’re implementing a short-term fix over the next 24–48 hours, which should eliminate the timeouts. A longer-term fix is also planned, but will roll out over several weeks, but early phases of that implementation should also reduce the request times that were originally triggering the 504 timeouts. What you can do: If you're currently affected, we recommend following updates here, and subscribing to the public issue: https://github.com/lemurheavy/coveralls-public/issues/1757 If your issue pattern differs from above, or you suspect a different root cause, reach out to [email protected], and we'll verify for you.

2. monitoring Sep 05, 2025, 04:05 PM UTC

   We are still working on a near-term fix. We will post here, and here when complete: https://github.com/lemurheavy/coveralls-public/issues/1757

3. monitoring Sep 08, 2025, 06:57 PM UTC

   All systems operational. Continuing to keep this open until we have released our short-term fix into production. Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757

4. monitoring Sep 09, 2025, 03:24 PM UTC

   All systems operational. We have released one of two parts of a near-term solution into production resolving a minority subset of 504 errors. We are still working on releasing part two into production. Subscribe for updates at this status page, or follow this public tracking issue for updates: https://github.com/lemurheavy/coveralls-public/issues/1757

5. monitoring Sep 12, 2025, 03:44 PM UTC

   All systems operational. Earlier today (6:45–7:45 AM PDT), we received elevated reports of 504 timeout errors. We have not been able to reproduce the issue since, but if you are still experiencing errors, please contact us at [email protected]. The affected areas may include: - Coverage Report Uploads (/api/v1/jobs) - Add Repos Page - Repo Page - Source File Page Fixes for the Add Repos, Repo, and Source File pages are scheduled to be deployed by end of day (PDT).

6. monitoring Sep 15, 2025, 06:57 PM UTC

   Mitigation in place. All systems operational. This morning we deployed additional capacity and autoscaling measures to reduce 504 errors on coverage report uploads: - Doubled our web server fleet (on top of the prior doubling when this issue began). - Enabled autoscaling at the web layer, allowing the fleet to double again automatically when NGINX response times exceed thresholds. The underlying trigger remains rare surges of upload requests from outlier repositories (750–1250 uploads per build). While we have paused processing for these repos, our HTTP servers must still handle the incoming requests until they stop. Timezone coverage: As a small team based in Los Angeles (PDT), our ability to respond in real time is most limited overnight (10p–6a PDT). Unfortunately, the primary outlier repos are in APAC, making this the window of highest risk. With these changes, we hope to reduce the occurrence of upload 504s during this window. We will monitor results closely and continue tuning autoscaling thresholds. Please let us know if you continue to see 504 errors on uploads.

7. monitoring Sep 22, 2025, 04:43 PM UTC

   Mitigated – Monitoring All systems operational. Recent mitigations, including fleet expansion and autoscaling, have reduced 504 timeout reports significantly. The remaining reports are infrequent and occur mostly during overnight and weekend hours (PDT). We are continuing to monitor closely and are working on a multi-part solution to eliminate all known causes. Until then, we are keeping this incident open in Monitoring. We will close it once 504 errors have returned to being unexpected, isolated events.

8. monitoring Sep 23, 2025, 07:14 PM UTC

   Fix for unrelated 500 errors: If you receive a `500` error with this error message format: > ⚠️ Internal server error. Please contact Coveralls team. Please know it is unrelated to the `504` errors being monitored in this open incident. Those, intermittent `500` errors are caused by a regression in one of the latest coverage-reporter releases: `v0.6.16` or `v0.6.17`. Workaround: Pin your coverage-reporter-version to `v0.6.15` in your integration config. For thorough instructions, see this public issue: https://github.com/coverallsapp/coverage-reporter/issues/180 We’re investigating the root cause and will post updates once a fix is released.

9. resolved Sep 25, 2025, 12:53 AM UTC

   500 Internal Server Errors on Uploads The recent 500 error surfacing during some coverage uploads as: > ⚠️ Internal server error. Please contact Coveralls team. has been resolved. A full postmortem will be published here soon. In the meantime, you can find more detail in the main tracking issue: https://github.com/coverallsapp/coverage-reporter/issues/180 Summary The root cause was ultimately infrastructure-related, not a regression in recent coverage-reporter releases. The previous workaround of pinning your coverage-reporter version is therefore not required. We have decided to close this incident, which we intentionally kept open for over a week to track a series of 504 and 5xx issues with overlapping root causes. In hindsight, the broadened scope made updates less clear than we'd hoped. With today’s resolution and the mitigations applied throughout the week, the occurrence of 504 errors during uploads (POSTs) has been significantly reduced. Going forward, any new 504 errors should be considered unexpected, isolated events. At the same time, we continue work on several instances of intermittent GET-related 504 errors affecting: - Source File pages - Repo pages - Add Repos pages Progress on those issues will be reported separately here: https://github.com/lemurheavy/coveralls-public/issues/1757

10. postmortem Sep 25, 2025, 04:35 PM UTC

    **This is a postmortem on this specific issue:** Intermittent 500 Errors on Coverage Uploads **Summary** Between September 20–24, some customers experienced intermittent `500 Internal Server Error` responses during coverage uploads \(`POST /api/v1/jobs`\). The issue was initially hard to diagnose because: * Failures did not surface reliably in our error tracker \(BugSnag\). * They appeared to affect only some requests, some customers. **Impact** * Some coverage uploads failed to process, causing build reporting delays or gaps. * Frequency was low enough to appear intermittent, which delayed detection and resolution. **Timeline** * **Sep 20–23**: First customer reports of intermittent 500s. Initial theories involved a regression in a recent release of our coverage-reporter integration \(client-side\). * **Sep 23–24**: Deep log analysis across ELB and application logs revealed errors concentrated on a single web server. * **Sep 24**: Confirmed that server alone was responsible for thousands of SSL-related failures \(`Faraday::SSLError`, `OpenSSL::SSL::SSLError`, `Seahorse::Client::NetworkingError`\). Other servers were clean. * **Sep 24**: Mitigation: that server was destroyed. Errors ceased immediately. **Root Cause** In terms of possible cause, we believe this additional Web server was provisioned during autoscaling with a different Ubuntu version than the rest of the fleet. This seemed to result in a broken or outdated CA certificate store, causing outbound SSL connections \(GitHub, Travis, etc.\) to fail intermittently—but then bubble up to a `500` error for the original request \(`POST` to `/api.v1/jobs`\). **Resolution** * Problematic server removed from service. * Future mitigation: verify baseline OS/version and CA store when adding new servers, especially via automation. * Next step: document the correct procedure to disable a single server in Cloud66 load balancers, instead of outright destroying, so we can retain the server for forensic investigations. **Lessons Learned** * Errors can hide if they don’t surface in the bug tracker. Direct log analysis is essential. * Even one misconfigured server can cause significant customer impact. * Consistency in OS/base image, and CA, is critical.

Read the full incident report →

Started Aug 20, 2025, 03:48 PM UTC· resolved Aug 20, 2025, 10:21 PM UTC

1. investigating Aug 20, 2025, 03:48 PM UTC

   We are currently investigating this issue.

2. investigating Aug 20, 2025, 05:11 PM UTC

   We are continuing to investigate this issue.

3. identified Aug 20, 2025, 05:11 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Aug 20, 2025, 07:08 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Aug 20, 2025, 10:21 PM UTC

   We have implemented an intermediate solution and believe this issue has been resolved for now. To fully resolve the root cause, we will need to implement a more long-term solution, which is currently in design. (Please read postmortem for more information.) In the meantime, we will be doing our best to monitor for spikes in traffic from outlier repos and manually respond where our intermediate solution may not mitigate as much as we hope it will.

6. postmortem Aug 20, 2025, 10:22 PM UTC

   We believe this issue has been resolved for now. The underlying cause still appears to be large spikes in incoming Web traffic from other outlier repositories that we have not yet identified or not yet paused. **Interim Solution**: To reduce the risk of recurrence, we have applied _temporary load balancer adjustments_ that change how requests are distributed, which should _lower_—if not _eliminate_—the frequency of **503** “**This website is under heavy load**” **errors**. **Permanent Solution**: We are also designing a permanent solution to _rate-limit abnormal request patterns_. This will require coordination at the policy/SLA level before it can be fully implemented. In the meantime, we will continue to closely monitor traffic and use targeted load balancer and web server configurations to mitigate the impact of outlier traffic spikes. **More details**: For a more detailed assessment / RCA of this incident and its recent, related incidents, see [this postmortem](https://status.coveralls.io/incidents/wqbsxnzv0jsf). **Update \(Thu, Aug 21\)**: We have identified a **different permanent solution**, which does not entail changes to SLA-level details for “outlier repos.” We may still implement such a solution, but our alternative solution should avoid further 503 errors and be implemented in the next 48-72 hrs.

Read the full incident report →

Started Aug 19, 2025, 03:21 PM UTC· resolved Aug 19, 2025, 05:26 PM UTC

1. investigating Aug 19, 2025, 03:21 PM UTC

   We are continuing to receive errors of customers receiving "This website is under heavy load" errors from our HTTP servers, even as traffic is normal. We implemented a fix last night that resolved the issue for 6-8 hrs, until we received a new report. We are investigating the issue to identify a permanent fix. In the meantime, if you receive this error, please re-try your request.

2. monitoring Aug 19, 2025, 04:24 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 19, 2025, 05:26 PM UTC

   We believe we have resolved all intermittent instances of the "This website is under heavy load" 503 error from our HTTP servers. If you should happen to receive that error from here, please let us know at: [email protected].

4. postmortem Aug 19, 2025, 06:28 PM UTC

   ### Postmortem: Reports of “Website under heavy load” errors We experienced multiple intermittent errors over the past several days before we were able to identify the true root cause and resolve the issue. **Root Cause** The errors were caused by a single outlier repository generating extremely high-volume requests \(750–1,800\+ coverage report uploads per build\). Combined with the default “sticky request” behavior in Passenger Enterprise \(which routes repeat requests from the same IP to the same HTTP server\), this overwhelmed individual servers. Once a server’s request queue was exhausted, subsequent requests returned a `503` error with the message: _“This website is under heavy load.”_ Although each server was able to process individual requests within normal timeframes, the concentrated traffic volume from a single repo and source IP could not be evenly distributed across servers. This led to repeated saturation of request queues and customer-visible errors. **Solutions Implemented** 1. We are testing new settings to override Passenger’s default “sticky request” behavior to allow requests to be distributed more evenly across servers. 2. We have paused processing for the outlier repository while we validate that the configuration changes are sufficient to prevent future incidents. **Next Steps** * Continue monitoring system performance to confirm stability. * Reintroduce the paused repository once we are confident the mitigations are effective. **Closing** We appreciate your patience as we worked through this issue. These changes are intended to permanently guard against similar incidents going forward. If you encounter unexpected errors, please contact us at [[email protected]](mailto:[email protected]). **Related incidents** 1. **Aug 13**: [Intermittent request rejections](https://status.coveralls.io/incidents/1n7plxrj8j44) 2. **Aug 14**: [Service unavailable with HTML error page or 500 errors](https://status.coveralls.io/incidents/v5mcbrsbhgt4) 3. **Aug 18**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/fr6sp5kyn128) 4. **Aug 19 \(Today\)**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/wqbsxnzv0jsf)

Read the full incident report →

Started Aug 14, 2025, 02:35 PM UTC· resolved Aug 14, 2025, 02:45 PM UTC

1. identified Aug 14, 2025, 02:35 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Aug 14, 2025, 02:45 PM UTC

   This incident has been resolved.

3. postmortem Aug 14, 2025, 02:47 PM UTC

   A postmortem for this incident and its related incidents has been posted [here](https://status.coveralls.io/incidents/wqbsxnzv0jsf): * [**Postmortem: Reports of “Website under heavy load” errors**](https://status.coveralls.io/incidents/wqbsxnzv0jsf) **Related incidents** 1. **Aug 13**: [Intermittent request rejections](https://status.coveralls.io/incidents/1n7plxrj8j44) 2. **Aug 14**: [Service unavailable with HTML error page or 500 errors](https://status.coveralls.io/incidents/v5mcbrsbhgt4) 3. **Aug 18**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/fr6sp5kyn128) 4. **Aug 19 \(Today\)**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/wqbsxnzv0jsf) ‌ **Previous postmortem, prior to final resolution:** We received customer reports early this morning PDT, Thu, Aug 14, indicating that the [Coveralls.io](http://Coveralls.io) service was unavailable for a substantial period overnight. While our Dequeue Graph appears unresponsive since 18:00 UTC yesterday Aug 13 / 11:00 PDT yesterday, Aug 13, this does not seem to be related to the unavailable period, which appears to have started around 02:00 UTC Aug 14 / 22:00 PDT Aug 13, because we were in service all day and afternoon yesterday, Wed, Aug 13, through EOD PDT. Our uptime ping did not catch this because Phusion Passenger was returning an HTML error page. Some customers report receiving a 500 error in that message; others report simply receiving HTML as their API response instead of JSON. While we have not seen production bugs specific to the issue, we suspect it’s because the issue occurred at the Passenger/Nginx layer, which simply prevented new traffic since occurring. As far as previously received coverage report uploads, we see successful background processing of jobs in queue all through the evening into this morning. We suspect as root cause a regression from a deploy around 5pm yesterday, due mainly to the fact that rolling back to a previous deploy has, so far, resolved the issue. We will investigate further to obtain a full understanding of the root cause, but in the meantime, we’ll continue monitoring. Please reach out if you are still experiencing any form of error page in Web requests, or any non-200 API response from C[overalls.io.](http://Coveralls.io) Our apologies for the inconvenience. **Note**: **Recommended** **Best Practice**: `continue-on-error: true` As a general best practice, we recommend adding the **input option** `continue-on-error: true` \(for [Coveralls GitHub Action](https://github.com/marketplace/actions/coveralls-github-action#inputs)\), or `continue_on_error: true` \(for [Coveralls Orb for CircleCI](https://circleci.com/developer/orbs/orb/coveralls/coveralls#commands-upload)\); or the `--no-fail` **flag** \(for [Coveralls Universal Coverage Reporter](https://github.com/coverallsapp/coverage-reporter?tab=readme-ov-file#usage)\), which will keep Coveralls errors from breaking your CI workflows in the future.

Read the full incident report →

Started Aug 13, 2025, 05:14 PM UTC· resolved Aug 13, 2025, 07:24 PM UTC

1. identified Aug 13, 2025, 05:14 PM UTC

   We've received reports that some used are receiving Web or API responses of: "This website is under heavy load (queue full)." We have identified the source of the issue and are working to resolve it now.

2. resolved Aug 13, 2025, 07:24 PM UTC

   Closing this incident as a potential false alarm. Connection stats show no issues and we cannot reproduce any responses of, "This website is under heavy load (queue full)." Nor do we see any in logs from this morning. We now believe the report we received this morning may have actually been encountered yesterday, around the same time as another report, before the incident was resolved. (Confirmed.) Please let us know if you have received the Web or API response, "This website is under heavy load (queue full)," and when you encountered it—which would be especially helpful.

3. postmortem Aug 19, 2025, 06:54 PM UTC

   A postmortem for this incident and its related incidents has been posted [here](https://status.coveralls.io/incidents/wqbsxnzv0jsf): * [**Postmortem: Reports of “Website under heavy load” errors**](https://status.coveralls.io/incidents/wqbsxnzv0jsf) **Related incidents** 1. **Aug 13**: [Intermittent request rejections](https://status.coveralls.io/incidents/1n7plxrj8j44) 2. **Aug 14**: [Service unavailable with HTML error page or 500 errors](https://status.coveralls.io/incidents/v5mcbrsbhgt4) 3. **Aug 18**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/fr6sp5kyn128) 4. **Aug 19 \(Today\)**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/wqbsxnzv0jsf)

Read the full incident report →

Started Aug 08, 2025, 08:00 AM UTC· resolved Aug 08, 2025, 03:00 PM UTC

1. resolved Aug 08, 2025, 07:34 PM UTC

   We experienced a period of elevated latency due to overnight traffic for about 6 hours today, Fri, Aug 8, from about 1am-7am PDT. We paused a couple of outlier repos, scaled servers to clear a backlog, and resolved ongoing latency. It took approximately 30-min to clear our remaining backlog of background jobs. We'd also like to take this opportunity to address a question we get, which is: Q: "If the Dequeue Time Graph is missing from the Status Page, does that mean processing has stopped?" A: No. While a missing graph can indicate a problem, it’s often isolated to the sampling process itself—either the cron job that collects the data or the single server we currently use for sampling. This “flagship” server runs additional packages and handles control and reporting tasks beyond normal app serving. At times, those extra duties can overload it, causing sampling failures without affecting overall build processing. We plan to broaden our sampling to multiple servers in the future.

Read the full incident report →

Started Jun 17, 2025, 01:34 PM UTC· resolved Jun 18, 2025, 03:37 AM UTC

1. identified Jun 17, 2025, 01:34 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Jun 17, 2025, 06:10 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 18, 2025, 03:37 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 03, 2025, 03:33 PM UTC· resolved Jun 04, 2025, 04:11 AM UTC

1. monitoring Jun 03, 2025, 03:33 PM UTC

   We are continuing to clear a backlog of background processing jobs for builds submitted in the past 18-24 hrs. While all systems are operational, there will continue to be latency on build times until we clear all background job queues, which are FIFO. Current estimate: 1-hour. We will post updates here until build times return to normal.

2. monitoring Jun 03, 2025, 05:34 PM UTC

   We took the following action to more quickly restore normal build times for all _new_ builds today: - We moved all unfinished background jobs from yesterday (Jun 2) into holding queues in order to restore normal build times for new builds from today (Jun 3). - We scaled resources to more quickly drain the existing queues of jobs from new builds from today (Jun 3). We will monitor progress on all new builds and provide updates here until we're fully caught up (zero (0) background jobs in queue). Thanks for your patience in the meantime as we restore the best possible performance to the service.

3. resolved Jun 04, 2025, 04:11 AM UTC

   All builds from today (Jun 3) have been processed. As background job queues cleared, build times returned to normal. We will continue monitoring.

Read the full incident report →

Started Apr 07, 2025, 05:37 AM UTC· resolved Apr 07, 2025, 09:14 AM UTC

1. investigating Apr 07, 2025, 05:37 AM UTC

   We are currently investigating this issue.

2. investigating Apr 07, 2025, 05:37 AM UTC

   We are continuing to investigate this issue.

3. identified Apr 07, 2025, 07:05 AM UTC

   The issue has been identified and a fix is being implemented.

4. identified Apr 07, 2025, 07:48 AM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Apr 07, 2025, 08:39 AM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring Apr 07, 2025, 08:52 AM UTC

   We are continuing to monitor for any further issues.

7. monitoring Apr 07, 2025, 09:01 AM UTC

   We are continuing to monitor for any further issues.

8. resolved Apr 07, 2025, 09:14 AM UTC

   This incident has been resolved.

9. postmortem Apr 11, 2025, 03:45 PM UTC

   ## **Coveralls Incident Postmortem** **Date of Incident:** April 6–7, 2025 **Published:** April 11, 2025 ### **Introduction** This postmortem outlines a recent incident that affected performance and stability for some of our customers. In the spirit of transparency and continuous improvement, we’re sharing what happened, what we learned, and how we’re moving forward. While our team has deep experience managing production infrastructure, we aren’t full-time PostgreSQL specialists. We’re full-stack developers — and this incident pushed us deeper into the internals of our database than we’d ever had to go before. It revealed blind spots, forced urgent decisions, and ultimately gave us a deeper understanding of what it will take to make Coveralls more resilient going forward. One of the nice things about developing an application for developers is that when we share our experience with technical challenges and lessons learned, we know many of our customers will relate. ### **Incident Summary** Between Sunday, April 6 \(PDT\), and early morning Monday, April 7 \(PDT\), Coveralls experienced performance degradation and elevated error rates affecting coverage reporting and related functionality across the platform. This culminated in a database overload and service outage from approximately 10:45 PM to 1:15 AM PDT. ### **Impact** * **Duration:** Approximately 8 hours of compounding service degradation, culminating in a full outage from ~10:45 PM Sun to 1:15 AM Mon PDT * **Affected Systems:** Coverage uploads, report generation, dashboard access, notifications delivery * **Customer Impact:** Delays in CI workflows, missing or partial coverage reports, stalled PR checks ### **Root Cause: Systemic Degradation** Roughly six weeks before this incident, we began seeing signs of database degradation — slower queries, rising table bloat, and an alarming acceleration in overall database size. Upon investigation, we realized that routine PostgreSQL maintenance \(VACUUM, ANALYZE, autovacuum\) — previously effective with our parameter tuning — was no longer working. This marked the beginning of a “runaway cycle,” where dead tuples piled up faster than the system could clear them, causing performance to degrade further, which in turn made optimizations even slower. In retrospect, we now understand this shift coincided with our database exceeding what we now recognize to be its _practical operational ceiling_ — a threshold around 50–60% of our 65TB AWS RDS instance size. While well below the official storage limit, this was the point at which PostgreSQL’s internal optimization routines began falling behind our write-heavy workload. Once we crossed it, most of the practices we had relied on for years to manage data growth and performance simply stopped working. In response, we began scheduling emergency optimization efforts across successive weekend maintenance windows. These included offline cleanup routines targeting bloated tables and long-running transactions, along with backups and preparatory work to reorganize our largest table partitions. The goal was to reduce database size while preserving historical data — and to free up space for optimizations that could no longer run reliably at our current scale. Unfortunately, one after another, those operations failed to complete in the allotted time — leaving us with fewer and fewer options as our database continued creeping toward the 65TB physical limit, and with it, the risk of out-of-disk errors and potential service failure. ### **Root Cause: Immediate Incident Trigger** The direct cause of the outage on April 6–7 was our decision to let three long-running `VACUUM FULL` operations continue on legacy partitioned tables after the end of our scheduled maintenance window. We had successfully reclaimed ~20TB of disk space earlier that day and expected the remaining operations — anticipated to reclaim an additional ~20TB — to finish within 1–2 hours based on size comparisons and previous completion times. Believing the risk was minimal — given the tables contained historical build data no longer accessed by active jobs or user queries — we reopened the application to production traffic. However, what we failed to anticipate was that: * **Background reconciliation jobs** still touched those locked tables indirectly * **PostgreSQL system processes** attempted metadata access that also triggered waits * These queries quickly **piled up behind table-level locks**, consuming all available connections This led to total connection saturation, application failures, and a full outage until the vacuum operations were forcibly terminated. ### **Outage Event: April 6–7 \(10:45 PM – 1:15 AM PDT\)** To reclaim space and restore performance, we began by decommissioning a set of legacy partitions containing historical build data no longer accessed by active workflows — a necessary step to reclaim ~20TB of space and allow critical optimization routines to proceed. We then launched nine parallel `VACUUM FULL` operations on bloat-heavy legacy tables. As the maintenance window ended, three of those operations were still running — expected to finish soon. Rather than risk losing the disk space gains by canceling mid-operation, we opted to let them finish. We added app-level protections to exclude the old data from queries and ran QA tests to ensure performance remained unaffected. Everything appeared stable: queries ran normally, monitoring showed no signs of blockage, and one engineer stayed online for two extra hours to confirm system health before calling it a night. At 10:45 PM, the site began returning 500 errors. Alerts indicated multiple servers were down, and the database was rejecting new connections with “Out of connections: Connections reserved for superuser.” RDS showed over 1,400 active connections — far beyond any level we’d previously seen in production. Most were tied up in blocked transactions: system-level activity and background reconciliation jobs that became stuck waiting on table locks held by the ongoing `VACUUM FULL` operations. We: * Entered Maintenance Mode * Killed all background jobs and blocked system queries * Canceled the remaining `VACUUM FULL` tasks This cleared the backlog and allowed the system to recover. The site was brought back online at 1:15 AM PDT and resumed normal operation. All remained well through 8:00 AM Monday PDT — the beginning of our regular Monday morning traffic spike. The system stayed stable and error-free under close watch. While a few queries ran slower than usual, autovacuum had resumed, and performance steadily improved over the following days as vacuum thresholds were tuned. ### **Lessons Learned** * **Legacy tables still matter:** Even unused partitions can be accessed by background queries or internal PostgreSQL processes * **Manual testing isn’t enough:** Latent issues from locked tables can take hours to appear under load * `VACUUM FULL` **must never run in production again**: It will only ever be used in offline maintenance contexts, and only when it can complete fully before reopening the application to production traffic \(We feel PostgreSQL DBAs nodding slowly with eyes closed at this one\) * **We didn’t realize we’d crossed the rubicon:** We now understand that for the way we were managing our schema, the real ceiling came far earlier than expected — around 50–60% of our RDS instance capacity. That’s where PostgreSQL’s behavior began to change, and where the practices we’d long relied on started to break down. * **Our tables are too big:** Our current manual partitioning scheme—organized by time—was not granular enough to prevent tables from growing to unmanageable sizes. Going forward, we’ll need to rethink partitioning entirely, possibly by table size, data volume, or much finer time intervals. * **We need to tier old data:** Partitioning alone isn’t enough. We need to offload older data to a different database instance or long-term storage tier — one designed for low-frequency access but with preserved integrity. * **We can't store everything in one place forever:** Coveralls has been storing every coverage report for every file in every commit for every repo it tracks for over 13 years — all in a single database instance. That approach hit its limit. That number now stands as a badge of our endurance — and a reminder of the architecture we must leave behind. ### **Remediation & Recovery** * All stuck optimization jobs were terminated * Background workers and queues were paused and restarted safely * Emergency disk space was reclaimed and preserved * Monitoring was expanded to detect and alert on blocked queries, autovacuum status, and connection saturation ### **Action Items** **Short-Term** * Establish hard guardrail against closing a maintenance window with open `VACUUM FULL` operations * Restrict application access to legacy tables not actively used in production workflows * Implement dashboards and alerts for `autovacuum` lag, blocked queries, and idle-in-transaction states * Tighten timeout thresholds to prevent query pile-ups during unexpected contention **Long-Term** * Complete staged migration to our new schema and infrastructure platform * Refactor partitioning to better manage table size and maintain performance * Build system-level resilience against lock pile-ups with improved observability, timeout strategy, and alerting * Design and implement a tiering strategy for long-term historical coverage data ### **Conclusion** This incident was a turning point. It taught us that the scale we’re operating at now requires a different mindset — one grounded in constraints we hadn’t fully appreciated before. When faced with a problem of scale and an urgent need to transition our infrastructure, we thought we had a clear destination in mind and rushed to get there — only to find the tracks couldn’t support the weight we were carrying. Now we understand this is a longer journey than we expected. We may not see the final station yet, but we know how to move forward: one stop at a time — making sure each station is stable, sustainable, and ready to serve our customers well before moving to the next. ### **Thanks** As mentioned above, no one on staff here is a PostgreSQL expert — and this incident forced us to learn more, and sooner, than we ever expected about running PostgreSQL at scale. We’d like to express our thanks to the many contributors and educators who share their hard-won PostgreSQL knowledge. Without their guidance, we couldn’t have learned what we now know—or recognize how much more there is to learn. * **Chelsea Dole**, of Citadel \(formerly of Brex\), for: * “[Postgres Table Bloat: Managing Your Tuple Graveyard](https://youtu.be/gAgbzvGT6ck?si=IY_O8vflQSToY3hn)” * “[It’s Not You, It’s Me: Breaking Up with Massive Tables via Partitioning](https://youtu.be/TafwSuLNxe8?si=Kk6L3rgZ5xTZQyMN)” * **Peter Geoghegan**, PostgreSQL contributor: * “[Bloat in PostgreSQL: a Taxonomy](https://youtu.be/JDG4bMHxCH8?si=_q-N2IsT2KiQRSPg)” * [His blog](https://pgeoghegan.blogspot.com/) * **Michael Christofides and Nikolay Samokhvalov**, of [Postgres.fm](https://postgres.fm/): * For their podcasts on Bloat, Index Maintenance, Out of Disk, and BUFFERS * Especially “[100TB and Beyond!](https://youtu.be/L6JWI296fyk?si=yQmPLykXR62YrXqb),” featuring Arka Ganguli \(Notion\), Sammy Steele \(Figma\), and Derk van Veen \(Adyen\)

Read the full incident report →

Started Mar 06, 2025, 03:48 PM UTC· resolved Mar 06, 2025, 04:17 PM UTC

1. monitoring Mar 06, 2025, 03:48 PM UTC

   Elevated latency during the US Eastern workday from ~7a-10a. We are clearing the associated backlog. ETA ~20-min.

2. resolved Mar 06, 2025, 05:17 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 03, 2025, 04:20 PM UTC· resolved Mar 03, 2025, 04:20 PM UTC

1. resolved Mar 03, 2025, 04:20 PM UTC

   Customers in the EU region experienced elevated latency during their core workday. We have updated our monitoring to to improve our response to elevated traffic that may increase latency in this region.

Read the full incident report →