Some services remain degraded including Insights reporting and intermittent image loading on Web Experience

Update timeline

1. investigating Oct 20, 2025, 01:18 PM UTC

   We are currently investigating an issue where Studio Insights Reports are not available on the Firstup US platform. Updates to follow asap.

2. investigating Oct 20, 2025, 01:37 PM UTC

   We are working with AWS to restore service to the Insights Reports. Initial investigation is pointing to a residual issue from the earlier Third Party issue with AWS.

3. investigating Oct 20, 2025, 02:26 PM UTC

   It has been confirmed that this is a residual issue from the AWS issue earlier and Engineering are working with AWS to resolve. It is not only Firstup impacted by this residual issue, Studio/ Web Experience and Mobile channels all remain available.

4. investigating Oct 20, 2025, 03:24 PM UTC

   AWS are still investigating the issue impacting Insights. There is also an issue impacting the display of some images on Web Experience and Campaigns sent, this is also third party caused and related to the AWS issue.

5. investigating Oct 20, 2025, 03:47 PM UTC

   We are continuing to investigate this issue.

6. investigating Oct 20, 2025, 04:56 PM UTC

   We are continuing to observe a low level of errors and intermittent network connectivity drops within the AWS cloud infrastructure. Service tuning on our side has been insufficient at times to handle the level of degredation we are observing with the underlying infrastructure. The following impact is being observed: * Web Experience - intermittent 502 Gateway Errors, blank white screen, errors loading content feed * Image transformation - some images are not displaying correctly and placeholders are being shown instead * User sync - we are reaching out directly to Customers we feel the user file upload may have failed while AWS S3 service wasn't available * Videos - AWS transcoding service appears to still be which is impacting video uploads

7. investigating Oct 20, 2025, 05:46 PM UTC

   We are continuing to observe a low level of errors and intermittent network connectivity drops within the AWS cloud infrastructure. Service tuning on our side has been insufficient at times to handle the level of degredation we are observing with the underlying infrastructure. The following impact is being observed: * Web Experience - intermittent 502 Gateway Errors, blank white screen, errors loading content feed * Image transformation - some images are not displaying correctly and placeholders are being shown instead * User sync - we are reaching out directly to Customers we feel the user file upload may have failed while AWS S3 service wasn't available * Videos - AWS transcoding service appears to still be which is impacting video uploads * Studio performance and availability degraded due to AWS backend network connectivity problems

8. monitoring Oct 20, 2025, 06:20 PM UTC

   We are observing substantial improvements in error rates platform wide. Moving incident to Monitoring status and Studio and Web Experience to fully operational. We have separate work streams for Insights, and web experience images intermittently not displaying.

9. monitoring Oct 20, 2025, 09:19 PM UTC

   We are continuing to monitor for any further issues.

10. monitoring Oct 20, 2025, 09:28 PM UTC

    Insights reporting on Firstup's US1 region remain unavailable while Amazon rebuilds the AWS Redshift data store that it relies on. We have received a number of progress updates from AWS throughout the day, but no firm commitment thus far of service restoration. All other Firstup services remain fully operational.

11. monitoring Oct 20, 2025, 09:50 PM UTC

    Insights reporting on the Firstup US1 region have now been restored and are fully operational. We will continue to keep all impacted services under monitoring for stability and functionality.

12. resolved Oct 24, 2025, 11:42 PM UTC

    AWS has remained stable and this incident is resolved.

13. postmortem Nov 03, 2025, 11:49 PM UTC

    ## Summary: On October 20th, 2025, at 12:05 AM PT, Firstup platform infrastructure monitoring tools fired multiple alerts of failed system health checks. Firstup engineers immediately acknowledged the alerts and started investigating. At 12:17 AM PT, the investigations revealed that both Studio and Employee Experience on the Firstup US platform were not accessible because of an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A platform incident was declared at 12:34 AM PT, and subsequently published on our [Status Page](https://status.firstup.io/incidents/26bv9gmp764g) at 12:44 AM PT. ## Severity: Sev-1 ## Scope: All customers on the Firstup US platform attempting to access Studio or the Employee Experience were in scope of this incident. ## Impact: Customers on the Firstup US platform primarily observed an error message while trying to log into or navigate within Studio or the Employee Experience. Some microservices on the Firstup US platform continued to exhibit degraded performance once access to Studio and Employee Experience was restored. These microservices included: · Insights Reports · Image/video uploading and processing · User Sync ## Root Cause: The root cause for this incident was determined to be an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A summary of the incident on the AWS platform can be found [here](https://aws.amazon.com/message/101925/). ## Mitigation: As AWS continued to work on the service disruption on their platform, access to Studio and Employee Experience on the Firstup US platform was restored by 2:37 AM PT, although some latency was still observed while navigating within these products. Some microservices on the Firstup US platform continued to experience intermittent degraded performance throughout the AWS incident, until it was fully resolved at 2:09 PM PT. ## Recurrence Prevention: In an effort to make the Firstup platform more resilient to an incident on a dependent third-party vendor such as AWS, we have adjusted our systems’ Health-checks and Liveliness probes to better react to pod slow startups which were observed to cause crash loops during the incident.