Cornerstone incident

Intermittent Errors on US PROD, PIL and STG environment - All US Swimlanes

Minor Resolved View vendor source →

Cornerstone experienced a minor incident on October 20, 2025 affecting Uptime and Uptime and 1 more component, lasting 6h 43m. The incident has been resolved; the full update timeline is below.

Started
Oct 20, 2025, 04:14 PM UTC
Resolved
Oct 20, 2025, 10:58 PM UTC
Duration
6h 43m
Detected by Pingoru
Oct 20, 2025, 04:14 PM UTC

Affected components

UptimeUptimeUptimeUptimeUptimeResponse TimeResponse TimeResponse TimeResponse TimeResponse Time

Update timeline

  1. identified Oct 20, 2025, 04:11 PM UTC

    Customers on US Swimlanes (all environments) may experience service degradation including Error messages and access issues due to the ongoing AWS US East Outage.

  2. identified Oct 20, 2025, 04:14 PM UTC

    Customers on US Swimlanes (all environments) may experience service degradation including Error messages and access issues due to the ongoing AWS US East Outage.

  3. identified Oct 20, 2025, 04:56 PM UTC

    Customers on US Swimlanes (all environments) may experience service degradation including Error messages and access issues due to the ongoing AWS US East Outage.

  4. identified Oct 20, 2025, 05:36 PM UTC

    We are engaged with AWS to track the mitigation efforts and recovery progress. We will continue to share updates as we learn more.

  5. monitoring Oct 20, 2025, 07:28 PM UTC

    AWS services are recovering, and impacted areas of our platform are stabilizing. Our teams continue to actively monitor system performance to ensure full restoration. We’ll provide additional updates as recovery progresses.

  6. monitoring Oct 20, 2025, 09:01 PM UTC

    AWS services are on track for full recovery. We continue to closely monitor progress until the incident is fully resolved.

  7. resolved Oct 20, 2025, 10:58 PM UTC

    AWS services are fully restored and after a period of monitoring we are considering this issue resolved.

  8. postmortem Nov 06, 2025, 07:37 AM UTC

    Incident Summary: On October 20th, 2025, clients hosted in the US PRD SL1,SL2,SL3,SL5,SL9 environment and connecting from the US-East-1 region experienced intermittent connectivity issues while accessing their portals. Affected users encountered HTTP 503 \(Service Unavailable\) errors during this period. ‌ Root Cause Analysis \(RCA\): The incident was caused by an outage experienced by Cornerstone’s CDN vendor. During the outage, certain CDN edge servers in the US-East-1 region were unable to establish connections with the origin servers, resulting in failed content delivery and 503 errors for end users attempting to access the service. ‌ Resolution: Engineering teams closely monitored the situation and collaborated with the CDN vendor to validate and confirm restoration of normal operations. Once the CDN provider resolved the underlying connectivity issue and edge nodes were fully reconnected to the origin servers, normal access to the portals was restored for all clients. ‌ Preventive Measures: * Vendor Coordination: Strengthen escalation and communication channels with CDN vendors to ensure faster visibility and resolution of regional outages. * Redundancy Planning: Evaluate multi-region and multi-vendor CDN configurations to reduce single-point dependency and improve service resilience.