Blues incident

Cellular notecards impacted by service provider outage

Resolved after 6 hours 48 minutes, affecting Cellular Notecards, on November 19, 2026.

resolved Nov 19, 2026, 09:38 PM UTC

Our cellular provider confirmed that an unexpected network outage occurred during maintenance in one of the two independent data centers that feeds Notehub. When that data center stopped forwarding traffic, it revealed a latent concurrency limitation in their alternate data center. The combination of the two issues caused intermittent Notecard connectivity. Our provider has committed to provide Blues with a root cause analysis. In addition, they are working to immediately correct the concurrency limit. We appreciate the trust you put in Blues. Outages, even partial ones, are not acceptable. In the coming days, we will share with you a root cause analysis and the mitigations we are implementing to prevent similar issues.

Started
Nov 19, 2026, 02:49 PM UTC
Resolved
Nov 19, 2026, 09:38 PM UTC
Duration
6h 48m
Detected by Pingoru
Nov 19, 2026, 02:49 PM UTC

Affected components

Cellular Notecards

Update timeline

  1. monitoring Nov 19, 2026, 02:49 PM UTC

    Between 02:31 and 11:40 UTC on Nov 19 (9:31pm to 6:40am EST) many cellular Notecards were unable to reach Notehub due to a partial outage with our cellular service provider. Full cellular connectivity has been restored. However, some Notecards may not reconnect until their penalty box timeout elapses. We are working with our cellular provider to identify the root cause. We will review their architecture to ensure that they have sufficient redundancy in place to prevent any impact to our customers should such partial outages recur in the future. In the meantime, we are continuing to monitor the situation closely.

  2. resolved Nov 19, 2026, 09:38 PM UTC

    Our cellular provider confirmed that an unexpected network outage occurred during maintenance in one of the two independent data centers that feeds Notehub. When that data center stopped forwarding traffic, it revealed a latent concurrency limitation in their alternate data center. The combination of the two issues caused intermittent Notecard connectivity. Our provider has committed to provide Blues with a root cause analysis. In addition, they are working to immediately correct the concurrency limit. We appreciate the trust you put in Blues. Outages, even partial ones, are not acceptable. In the coming days, we will share with you a root cause analysis and the mitigations we are implementing to prevent similar issues.

Your product runs on more than Blues

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →