Squirrel Cloud incident

Production Outage due to file server upgrade from Microsoft

Squirrel Cloud experienced a notice incident on January 28, 2022 affecting API and HUB, lasting 20m. The incident has been resolved; the full update timeline is below.

Started: Jan 28, 2022, 03:00 AM UTC
Resolved: Jan 28, 2022, 03:20 AM UTC
Duration: 20m
Detected by Pingoru: Jan 28, 2022, 03:00 AM UTC

Affected components

APIHUB

Update timeline

resolved Feb 07, 2022, 12:13 PM UTC

The Production environment went offline around 7:00 PM PST 01/27 and required a restart of the Hub & API services to become operational again. During this time, FOH terminals went into offline mode and the BOH was inaccessible. Everything became functional again shortly after the services were restarted at 7:20 PM PST. This issue is resolved now and it is caused due to file server upgrade by Microsoft which made the platform inaccessible for a few mins. Restarting of services brought them back to operational. No components were marked as affected by this incident.
postmortem Feb 07, 2022, 12:14 PM UTC

The application was restarted due to storage failures in the web app at 03:00 AM UTC and completed at 03:01 AM UTC but the application went into hung state and was failing with exceptions which we call it as startup failures caused the web application downtime. Once the web app was restarted manually at 03:20 AM UTC the thread was released and the issue was resolved. This issue is caused due to internal file server upgrade by Microsoft