Third Party Service AWS issue causing Studio and Web Experience to be unavailable

Update timeline

1. investigating Oct 20, 2025, 07:44 AM UTC

   Both Web Experience and Studio are currently down for US customers, EU Web Experience and Studio are available. The issue is linked to a broader AWS outage impacting US customers. We have reached out to AWS for updates.

2. investigating Oct 20, 2025, 07:57 AM UTC

   Update provided by AWS: We can confirm increased error rates and latencies for multiple AWS Services in the US. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause. We will provide an update in 45 minutes, or sooner if we have additional information to share.

3. identified Oct 20, 2025, 09:07 AM UTC

   Update from AWS Oct 20 2:01 AM PDT We have identified a potential root cause for error rates.... Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.... We will continue to provide updates as we have more information to share, or by 2:45 AM.

4. identified Oct 20, 2025, 09:33 AM UTC

   Update provided by AWS: We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

5. identified Oct 20, 2025, 09:37 AM UTC

   Services are returning to normal after the latest update provided by AWS. Both Studio and Web Experience and all other channels are available. We are performing checkouts on all services. StatusPage will be set to Monitoring until AWS advises this issue has been fully resolved.

6. resolved Oct 20, 2025, 12:07 PM UTC

   Checkouts complete and no further reports from AWS in relation to any downtime expected. Root Cause Analysis will be published when provided by AWS.

7. postmortem Nov 03, 2025, 11:46 PM UTC

   ## Summary: On October 20th, 2025, at 12:05 AM PT, Firstup platform infrastructure monitoring tools fired multiple alerts of failed system health checks. Firstup engineers immediately acknowledged the alerts and started investigating. At 12:17 AM PT, the investigations revealed that both Studio and Employee Experience on the Firstup US platform were not accessible because of an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A platform incident was declared at 12:34 AM PT, and subsequently published on our Status Page at 12:44 AM PT. ## Severity: Sev-1 ## Scope: All customers on the Firstup US platform attempting to access Studio or the Employee Experience were in scope of this incident. ## Impact: Customers on the Firstup US platform primarily observed an error message while trying to log into or navigate within Studio or the Employee Experience. Some microservices on the Firstup US platform continued to exhibit degraded performance once access to Studio and Employee Experience was restored. These microservices included: · Insights Reports · Image/video uploading and processing · User Sync ## Root Cause: The root cause for this incident was determined to be an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A summary of the incident on the AWS platform can be found [here](https://aws.amazon.com/message/101925/). ## Mitigation: As AWS continued to work on the service disruption on their platform, access to Studio and Employee Experience on the Firstup US platform was restored by 2:37 AM PT, although some latency was still observed while navigating within these products. Some microservices on the Firstup US platform continued to experience intermittent degraded performance throughout the AWS incident, until it was fully resolved at 2:09 PM PT. ## Recurrence Prevention: In an effort to make the Firstup platform more resilient to an incident on a dependent third-party vendor such as AWS, we have adjusted our systems’ Health-checks and Liveliness probes to better react to pod slow startups which were observed to cause crash loops during the incident.