IT Portal incident

US Cloud Services Down

Notice Resolved View vendor source →

IT Portal experienced a notice incident on December 17, 2025 affecting US API and US Cloud, lasting 52m. The incident has been resolved; the full update timeline is below.

Started
Dec 17, 2025, 09:53 AM UTC
Resolved
Dec 17, 2025, 10:46 AM UTC
Duration
52m
Detected by Pingoru
Dec 17, 2025, 09:53 AM UTC

Affected components

US APIUS Cloud

Update timeline

  1. monitoring Dec 17, 2025, 09:53 AM UTC

    SQL Services affecting central login was down after updates. Issues reconnecting encrypted volumes after reboot. Volumes were manually reconnected and services are operational.

  2. resolved Dec 17, 2025, 10:46 AM UTC

    This incident has been resolved.

  3. postmortem Dec 22, 2025, 01:09 PM UTC

    **Incident Summary:** SQL Services affecting central login experienced downtime following routine updates. Issues were encountered when reconnecting encrypted volumes after system reboot. **Root Cause:** A recent Windows update modified the default behavior of PowerShell's `Invoke-WebRequest` cmdlet. Our automated scripts that handle volume reconnection were executing without the `-UseBasicParsing` parameter, causing them to pause and wait for user confirmation instead of executing non-interactively. **Resolution:** Encrypted volumes were manually reconnected to restore service availability. Our automation startup scripts have been updated to include the `-UseBasicParsing` parameter in all `Invoke-WebRequest` calls, ensuring scripts execute without requiring user interaction. **Preventive Measures:** * All PowerShell automation scripts have been audited and updated with the `-UseBasicParsing` parameter * Testing procedures will be enhanced to validate script execution in non-interactive environments following system updates * Additional monitoring has been implemented to detect script execution failures more rapidly **Timeline:** * 4:53 PM EST - Issue detected, services down * Manual intervention initiated * 5:46 PM EST - Services restored, incident resolved