Hologram incident

Service degradation on some G2 (89357*) profiles

Minor Resolved View vendor source →

Hologram experienced a minor incident on November 14, 2025 affecting Global Cellular Data Network and SMS over cellular - Device Terminated and 1 more component, lasting 11d 6h. The incident has been resolved; the full update timeline is below.

Started
Nov 14, 2025, 04:28 PM UTC
Resolved
Nov 25, 2025, 10:28 PM UTC
Duration
11d 6h
Detected by Pingoru
Nov 14, 2025, 04:28 PM UTC

Affected components

Global Cellular Data NetworkSMS over cellular - Device TerminatedSMS over cellular - Device Originated

Update timeline

  1. investigating Nov 14, 2025, 04:28 PM UTC

    We are currently investigating an issue where a small number of G2 profiles are reported to have issues when trying to open new sessions.

  2. investigating Nov 18, 2025, 03:13 PM UTC

    We are continuing to investigate this issue.

  3. identified Nov 18, 2025, 06:13 PM UTC

    We’ve identified the issue. An abnormal spike in signaling activity on our US PGW for G2 profiles (89357*) is causing intermittent failures when devices attempt to establish new sessions. During these spikes, the PGW is running out of available resources, which leads to session setup attempts failing. We’re working closely with our upstream partners to expand capacity on the PGW and put additional stabilizing measures in place. We’ll share further updates as this work progresses.

  4. identified Nov 20, 2025, 06:18 PM UTC

    Earlier today, a fix was deployed and successfully validated in isolation, but once rolled out system-wide it introduced instability and had to be reverted. Our upstream partner have since completed the adjustments needed, and the updated fix is now scheduled for deployment tomorrow, November 21st at 10:00 AM UTC+2. What to expect: During this maintenance window, all SIMs routed through our US (Miami) PGW will briefly disconnect as sessions are reset. Devices should automatically re-establish connectivity once the change is applied. We’ll post a final update once the fix is live and the PGW is back to normal operation.

  5. monitoring Nov 21, 2025, 02:33 PM UTC

    The upstream fix has now been deployed, and early checks look good. We’re monitoring session behavior closely as devices cycle through to make sure everything stays stable. We’ll post another update once the rollout is complete and validated.

  6. monitoring Nov 22, 2025, 12:02 PM UTC

    Last night, we observed another spike in disconnection events on the US PoP PGW affecting some G2 (89357*) profiles. The responsible core network engineering team worked through the night and will apply an additional patch required to fully stabilize session handling. To apply this fix, the PGW is being restarted early this morning. During the restart window, affected devices may briefly disconnect, but sessions should automatically recover once the patch is applied and the PGW is back online. We’re continuing to monitor session establishment rat es and disconnection counts closely as the platform comes back up. A further update will follow once stability is confirmed.

  7. monitoring Nov 22, 2025, 06:42 PM UTC

    Our upstream partner has completed the fixes related to the recent service degradation affecting a subset of G2 (89357*) profiles routed through the US (Miami) PGW. Early results look stable, and session behavior has returned to expected levels. We’re continuing to monitor the PGW closely as devices reconnect and cycle through normal attach patterns. A small number of SIMs might still be impacted by provisioning inconsistency caused by the incident. The majority of these have already been corrected, and the remaining affected SIMs are being processed and validated. This work will continue through the weekend and into next week, but the overall impact is now very limited and contained. No action is required from customers at this time. Devices should reconnect automatically. If any individual SIMs continue to show abnormal behavior, please open a ticket so we can verify and remediate them. We’ll post another update once monitoring is fully complete.

  8. resolved Nov 25, 2025, 10:28 PM UTC

    All systems are stable and operational