Snow Atlas - Australia - Few SAM core pages are not loading

Update timeline

1. investigating Jun 12, 2026, 01:38 AM UTC

   Incident Description: We are investigating an issue affecting a small subset of customers in the Australia production environment using Snow Atlas. Affected users may be unable to access certain Snow Atlas pages, including the Home page and SAM Core Overview page. Priority: P2 Restoration Activity: Technical teams have engaged and are actively investigating the issue. Current analysis confirms the impact is limited to a small subset of customers in the Australia region. We are closely monitoring the environment and will provide further updates as additional information becomes available.

2. resolved Jun 12, 2026, 06:24 AM UTC

   Our teams identified an issue affecting the database server and performed a restart, which successfully restored service. We will continue to investigate the underlying root cause to determine what led to the issue and identify any necessary preventative measures. Additional details and findings will be shared in a post-mortem report.

3. postmortem Jun 12, 2026, 02:38 PM UTC

   **Description:** Snow Atlas – APAC – Login Failures and HTTP 500 Errors **Timeframe:** May 28, 2026, 20:57 PDT to May 29, 2026, 03:32 PDT ‌ **Incident Summary** ‌ On Thursday, May 28, 2026, at 20:57 PDT, an incident was identified impacting a subset of Snow Atlas customers in the APAC production environment. Affected users were unable to access the platform or experienced degraded application functionality. This included HTTP 500 and 404 errors, as well as instances where pages loaded incompletely with no data before ultimately returning service unavailability messages. Technical teams engaged immediately and identified degraded performance within a core routing component responsible for handling traffic. As the issue progressed, backend services were unable to communicate reliably, leading to increased request failures and overall application instability for impacted tenants. Initial mitigation actions, including targeted service restarts, successfully restored access for affected customers. Teams continued to investigate the underlying cause while maintaining enhanced monitoring to validate platform stability. A subsequent controlled restart and recovery of the impacted routing services fully resolved the contention conditions. Following these actions, no further customer impact was observed. The platform remained stable throughout the monitoring period, and the incident was formally resolved after normal service operations were confirmed. ‌ **Root Cause** ‌ The incident was triggered by an automated infrastructure upgrade initiated by the cloud service provider. This upgrade resulted in a sequential restart of infrastructure nodes supporting the Snow Atlas platform, causing a large number of application services to restart within a compressed timeframe. During the recovery phase, a critical routing component responsible for service discovery and request routing became saturated due to the high volume of service registration activity. Consequently, routing information was intermittently unavailable to dependent services, leading to application errors, request failures, and overall service instability for a subset of APAC customers. Contributing Factors: * The provider-initiated infrastructure upgrade was unplanned from an application perspective and triggered a large-scale, simultaneous service restart across the environment. * The routing service had limited redundancy, increasing the potential for service disruption under certain conditions. * During recovery, the temporary unavailability of the routing service prevented dependent services from resolving required routes, resulting in HTTP 404 and HTTP 500 errors across multiple application functions. * Elevated recovery activity, combined with the lack of redundancy, amplified both the impact and duration of the incident. ‌ **Remediation Actions** ‌ The following remediation steps were implemented to restore service functionality: * Technical teams identified the affected routing component and performed targeted recovery actions to restore service. * Service restarts were completed to re-establish routing functionality and stabilize affected platform services. * A controlled restart procedure was executed to ensure clean recovery of the impacted components. * Enhanced monitoring was maintained throughout the recovery process to validate platform stability and detect any recurrence. * Root cause analysis was completed and corrective actions were implemented to address the underlying infrastructure and resiliency gaps. ‌ **Future Preventative Measures** ‌ * Controlled Infrastructure Maintenance - Automatic upgrades have been disabled for the affected environment. Future infrastructure upgrades will be performed through planned maintenance activities with appropriate scheduling, oversight, and validation. * Increased Service Redundancy - The routing service has been scaled up to run multiple replicas across separate infrastructure nodes. * Improved Platform Resiliency - Platform resiliency controls have been reviewed and enhanced to better handle large-scale service restart events and reduce the risk of service discovery or routing bottlenecks during recovery operations. * Enhanced Monitoring and Early Detection - Additional monitoring and alerting have been implemented to provide earlier visibility into service degradation and infrastructure events that could impact platform availability.