Firstup Outage History

Started Oct 06, 2026, 08:20 PM UTC· resolved Oct 06, 2026, 10:47 PM UTC

1. investigating Oct 06, 2026, 08:20 PM UTC

   We are currently investigating an elevated error rate when loading employee experience causing slow page loads and intermittent timeouts.

2. identified Oct 06, 2026, 08:24 PM UTC

   Root cause has been identified. Impact on US1 region should be mitigated, and full recovery in process. US2 and EU regions are being investigated now.

3. identified Oct 06, 2026, 08:32 PM UTC

   EU region impact mitigated and fully recovered. Continuing to monitor region health. Work continues to mitigate impact on US2.

4. identified Oct 06, 2026, 09:00 PM UTC

   New user sessions cannot be created still on US2 region, but other services have recovered. Users with existing active sessions should be able to access and interact with programs hosted on that region.

5. monitoring Oct 06, 2026, 09:15 PM UTC

   Impact on US2 has been mitigated, and remaining services are fully operational. Actively monitoring all regions to ensure steady state.

6. monitoring Oct 06, 2026, 09:37 PM UTC

   Monitoring has confirmed all regions remain health, and the incident has now been fully resolved.

7. resolved Oct 06, 2026, 10:47 PM UTC

   This incident has been resolved.

8. postmortem Oct 14, 2026, 09:13 PM UTC

   **Summary:** On October 6th, 2026, at 1:00 PM PT, our monitoring alerted us to a high number of connection errors in the production environments shortly after a hotfix was deployed. Preliminary engineering investigations coupled with internal and external reports of platform loading errors confirmed platform reachability issues. A platform incident was declared at 1:11 PM PT, and the hotfix was rolled back, along with a targeted code fix deployment to address the underlying code error. All services were fully restored by 2:09 PM PT the same day. **Severity:** Sev-1 **Scope:** All Firstup customers were affected, with customers in US2 and EU platforms experiencing a more prolonged service disruption due to circular restart dependencies that had to be addressed separately from the hot fix rollback. Customers in the US1 platform experienced a brief disruption of service primarily accompanied by degraded application responsiveness. **Impact:** * **US1:** From 1:00 PM PT to 1:19 PM PT, service was degraded and the platform processed up to 70% less work than normal. * **US2:** From 1:00 PM PT to 2:09 PM PT \(69 minutes\), users couldn't log in to Studio, Employee Experience or the support portal, and extended message delivery times. * **EU:** From 1:00 PM PT to 1:26 PM PT \(26 minutes\), users had the same problems as in US2. **Root Cause:** The root cause for this incident was identified to be the deployment of a hotfix that was built on an updated shared base software image. That image included newer networking and encryption libraries that don't work with the web server setup of one of our core services, so the service crashed. While this problem had already been identified and protected against in our regular release processes, it had not been protected against in the hotfix deployment process branch. Our systems and services health checks didn't catch the failure until the healthy copies of the impacted service had already been replaced. Additionally, startup dependencies between two backend services increased the time to recovery and was more pronounced in the US2 platform. **Mitigation:** In US1 and EU, the deployed hotfix was rolled back and services were restored by 1:19 PM PT and 1:26 PM PT, respectively. In US2, and in addition to the hotfix rollback, we deployed a code fix that: * removed the startup dependency between the two backend services * locked the base software image to the last working version This fully restored service by 2:09 PM PT. The same fix then went to all regions in a scheduled release at 6:50 PM PT on October 6th, 2026, with no customer impact. **Recurrence Prevention:** To prevent this incident from happening again due to the same root cause, Firstup has done \(or committed to doing\) the following: * Lock and version shared base software images in every build process, including hotfixes, so that what is test is exactly what is released. * Roll out changes in stages, starting with one region or part of capacity, so problems are caught before they reach every region. * Strengthen health checks so they confirm the service is actually working before it takes traffic. * Removed the identified startup dependencies between the two core services in the US1 and EU platforms by deploying the code fix during our maintenance window on October 6th, 2026. * Conduct a review of any remaining dependencies for similar risks. * Tighten the hotfix process. Hotfixes that aren't deployed promptly will need re-approval, and related changes across services will be deployed together. * Update monitoring and recovery runbooks in every production region.

Read the full incident report →

Started Oct 01, 2026, 08:40 PM UTC· resolved Oct 01, 2026, 10:24 PM UTC

1. investigating Oct 01, 2026, 08:40 PM UTC

   We are currently investigating multiple reports of shortcuts not rendering as expected for programs hosted on US2. No other regions are impacted at this time.

2. identified Oct 01, 2026, 08:48 PM UTC

   We have identified a potential root cause for this issue and are working to mitigate it. Another update will be provided within 1 hour.

3. monitoring Oct 01, 2026, 09:07 PM UTC

   A fix for this issue has been released. A refresh of the Employee Experience page may be required to reload the shortcuts. We will be placing the impacted components under monitoring for now.

4. resolved Oct 01, 2026, 10:24 PM UTC

   This incident has been resolved, and all impacted components have remained stable and fully available. A Root Cause Analysis for this incident will be provided as soon as the incident postmortem has been concluded.

Read the full incident report →

Started Sep 09, 2026, 02:06 PM UTC· resolved Sep 09, 2026, 09:10 PM UTC

1. investigating Sep 09, 2026, 02:06 PM UTC

   We are currently investigating reports where Insights Reports are intermittently failing to load for EU customers only. We will provide an update within 1 hour.

2. identified Sep 09, 2026, 03:00 PM UTC

   We have identified a potential root cause for this issue and are working towards mitigation. Another update will be provided within the next 1 hour.

3. monitoring Sep 09, 2026, 04:00 PM UTC

   We have fixed the underlying issue with loading Insights Reports and completed a data backfill task to restore any data that may have been missed in the reports. We will be placing the affected components under monitoring for now.

4. resolved Sep 09, 2026, 09:10 PM UTC

   Insights Reports have remained stable and fully available after the fix was released in the production environment. This incident is now resolved. We will publish a Root Cause Analysis here once a full incident postmortem has been completed.

Read the full incident report →

Started Aug 13, 2026, 05:06 PM UTC· resolved Aug 14, 2026, 02:31 AM UTC

1. investigating Aug 13, 2026, 05:06 PM UTC

   We are currently investigating a delay in data populating in Insights reports. Reports in scope include the Campaign Results Page, Campaign Performance, Videos overview, hover popup modal, Campaign Delivery, and possibly others.

2. identified Aug 13, 2026, 05:59 PM UTC

   The issue has been identified that has caused a backup of data pipeline processing and are in the process of working through the data backfill. We expect reporting data to be fully caught up within the next 2 hours with incremental improvements already visible now.

3. monitoring Aug 13, 2026, 06:59 PM UTC

   We are continuing to actively monitor the data backfill progress and remain on target for full restoration of normal processing times by 1PM PT. Only programs on the US1 region are in scope of the issue.

4. monitoring Aug 13, 2026, 08:15 PM UTC

   There have been a few unexpected slowdowns in the backfill processing and we now expect the completion to be at least another hour from now. Also, not originally considered in scope, Poll reporting is also affected and being backfilled along with other reporting data.

5. monitoring Aug 13, 2026, 09:39 PM UTC

   All processing for events that occurred on 8/12 has concluded. There are events on 8/13 that are still being processed as that backfill is competing against new events coming in today. As a result, the backfill processing for today's events has had to be restarted several times-- adding unexpected delay to the completion time. Next update will be 1 hour from now, or following the next milestone completion, whichever is sooner.

6. monitoring Aug 13, 2026, 10:41 PM UTC

   Backfill process continuing to run for 08/13 events. Blocker has been removed that required multiple restarted and we have more confidence now in the throughput of the backfill process that will be complete for 08/13 within the hour. Focus will then shift to historic events on 08/10 and 08/11 which are also in scope.

7. monitoring Aug 14, 2026, 12:38 AM UTC

   Backfill for 08/13 complete and we are now back at normal operating cadence for new data being ingested. Dashboards are all up to date, and campaign delivery reports, and hoverover reports from the campaign results page are in the process of being refreshed.

8. resolved Aug 14, 2026, 02:31 AM UTC

   This incident is now resolved and all affected components marked fully operational.

9. postmortem Sep 14, 2026, 05:16 PM UTC

   **Summary:** On August 12th, 2026, a system monitor alerted us of a potential issue within our metrics data processing pipelines. Preliminary engineering investigations coupled with internal and external reports of an issue with Insights Reports revealed a wider scope and impact. A platform incident was declared on August 13th, 2026, at 16:33 UTC, and an incident response team was assembled. After determining the cause of the unprocessed data backlog and mitigating its impact, data backfill resumed on August 13th, 2026, at 23:22 UTC, eventually completing, and service was fully restored on August 14th, 2026, at 02:31 UTC. **Severity:** Sev-2 **Scope:** Between August 12th, 2026, at 10:00 UTC and August 14th, 2026, at 02:31 UTC \(40 hours and 31 minutes\), Studio users in the Firstup US1 environment were unable to reliably utilize Insights Reports. Users in the US2 and EU environments were not in the scope of this incident. **Impact:** Studio users in the Firstup US1 environment observed loading issues and stale data in Studio Insights Reports. Insights Reports in the US2 and EU environments were not impacted by this incident. **Root Cause:** The primary root cause was identified to be a software defect due to missing input data validation from the Employee Experience \(EE\) client side to the data ingestion layer. The client-side _time\_spent\_reading_ property accumulated active user reading duration without a cap or periodic flush, emitting only when the page/article closed. Any user reading session remaining continuously active for ~24.86 days produced a value exceeding the allowed integer boundary value, causing an integer overflow to downstream data pipelines that were not designed to handle integer overflows. As a resulted, the data pipelines failed, causing Insights Reports to stop ingesting new metrics. **Mitigation:** To stabilize and remediate the data pipeline, the following steps were taken: * **Truncation at Ingest:** A truncation filter was applied at the ingestion layer to cap incoming _time\_spent\_reading_ values, halting new integer overflows from reaching downstream pipelines. * **Data Sanitization:** Existing overflow values in _real\_time\_events\_details_ were manually reduced to allow transformation pipelines to process records. * **Lock Resolution:** An underlying data table lock on improperly terminated processes was identified and cleared, allowing frozen backfills to resume. * **Backfill & Schedule De-confliction:** Automated scheduled tasks that were running concurrently with the manual backfill scripts were resolved, allowing campaign metrics and integration data backfills to complete successfully. **Recurrence Prevention:** To prevent recurrence and improve resilience, Firstup has committed to the following actions: * **Input Sanitation & Capping:** Implement input restrictions on _time\_spent\_reading_ events from the Employee Experience \(EE\) client side, and on the data ingestion layer to cap values to 1 hour \(3,600,000 ms\) before they enter the data processing pipelines. * **Playbook Expansion:** Update the Engineering Playbook with explicit guidance on identifying/clearing process locks, disabling scheduled jobs during manual backfills, and running backfill scripts. * **Pipeline Resilience & Graceful Error Handling:** Evaluate per-record error handling and modular pipeline refactoring to prevent an isolated invalid record from halting global pipeline processing across all customer programs.

Read the full incident report →

Started Jun 22, 2026, 05:11 PM UTC· resolved Jun 23, 2026, 01:41 AM UTC

1. investigating Jun 22, 2026, 05:11 PM UTC

   We are investigating reports where campaign emails are not being delivered to the entire targeted campaign audience. We will provide you with an update within 1 hour.

2. identified Jun 22, 2026, 05:48 PM UTC

   We have identified a code change that resulted in the observed behavior. We are currently rolling back that code change and will provide another update once that is completed.

3. identified Jun 22, 2026, 06:51 PM UTC

   We are still in the process of rolling back the code change we believe caused the observed behavior. We anticipate this should be completed soon, but we will provide another update on its progress within the next hour.

4. identified Jun 22, 2026, 07:11 PM UTC

   We currently estimate that the code change rollback will be completed within the next 2 hours, which will fully mitigate the impact of the observed behavior. We will provide another update as soon as this is completed.

5. identified Jun 22, 2026, 08:09 PM UTC

   We have completed the code change rollback, and this incident is now mitigated. We will be placing the impacted component(s) under active monitoring to ensure continued proper functionality.

6. monitoring Jun 22, 2026, 08:09 PM UTC

   A fix has been implemented and we are monitoring the results.

7. resolved Jun 23, 2026, 01:41 AM UTC

   This incident is now resolved and all components marked fully operational. No further issues observed while monitoring since the platform incident was mitigated at 1PM PT today.

8. postmortem Jul 01, 2026, 04:37 PM UTC

   ## Summary: On June 22nd, 2026, we started receiving reports where email campaigns were being delivered to only a subset of the targeted email audience. Following initial triaging and troubleshooting, a platform incident was declared, and the incident response team began looking at the reported issue. The unexpected behavior was determined to be due to a logic error in user reachability through the email channel, and the software change implicated was immediately rolled back to resolve the issue. ## Severity: Sev-2 ## Scope: Some campaigns sent between 12:54 AM PT on June 20th, 2026, and 12:59 PM PT on June 22nd, 2026, to an email audience from the Firstup platform. ## Impact: Emails were not sent to the entire email audience for campaigns sent during the incident duration. ## Root Cause: The root cause for this incident was determined to be a regression from a software change that was made on June 20th, 2026, to align with the reachability definitions that we introduced in the Q2 quarterly release. A software bug was identified that returned a “Unknown” email address reachability status when an email audience was being processed for campaign delivery in the backend, resulting in those email addresses being excluded from the campaign’s delivery backend path. ## Mitigation: To immediately mitigate this incident, a rollback of the software change was performed, which was completed at 12:59 PM PT, restoring the reachability data to its previous state platform-wide. For customers who had campaigns in scope of this incident, and needed to send the email to users who were initially excluded from the campaign’s delivery, our Support team provided a list of those users to be retargeted separately via a duplicate campaign. ## Recurrence Prevention: Our recurrence prevention measures include: * Investigate and fix the software bug that returned the “Unknown” email address reachability status. * Update our QA processes to include robust end-to-end testing on all New Product Features and Enhancements before being released.

Read the full incident report →

Started May 07, 2026, 05:03 PM UTC· resolved May 07, 2026, 07:13 PM UTC

1. investigating May 07, 2026, 05:03 PM UTC

   We identified an underlying issue where the Comments card in the Campaign Performance Report was not consistently displaying the correct data. As a precaution, this card was temporarily disabled while our Engineering team investigated and resolved the root cause. This step was taken to ensure data accuracy and integrity.

2. monitoring May 07, 2026, 05:04 PM UTC

   The Comments card’s functionality has now been restored and is fully functional.

3. resolved May 07, 2026, 07:13 PM UTC

   Our monitoring indicates that the Comments card in the Campaign Performance Report continues to function as expected. This incident is now resolved.

4. postmortem Jun 05, 2026, 06:34 PM UTC

   ## Summary: On May 7th, 2026, we received reports that Studio users were able to see campaign titles, user names and email addresses, as well as comments made, that were not part of their program within the “Latest Comments” card or the Campaign Performance report. Following initial triaging and troubleshooting, a platform incident was declared, and the incident response team began looking at the reported issue. ## Severity: Sev-2 ## Scope: Any Studio user on the Firstup platform who accessed the Campaign Performance report directly from the Insights landing page, without the Content\_Id filter enabled. ## Impact: Studio users on the Firstup platform were able to see and download Personal Identifying Information \(PII\) for users outside their own program via the “Latest Comments” card in the Campaign Performance report. The exposed PII and other program information included: · Commenter’s first and last names · Commenter’s email address · Campaign publisher’s first and last name · Any PII contained within the raw comment text itself · Program ID ## Root Cause: The root cause for this incident was determined to be a regression from a configuration change that was released on April 24th, 2026, as part of a bug fix, which changed the format of the program\_id parameter, effectively removing the program\_id filter within the “Latest Comments” card, if a campaign\_id was not selected. However, if a campaign\_id was selected, the “Latest Comments” card would also filter to the campaign’s corresponding program\_id. ## Mitigation: To immediately mitigate data exposure, the “Latest Comments” card was reconfigured to return no results at 9:23 AM PT. A fix for the root cause was developed, tested, and released at 9:50 AM PT, and the “Latest Comments” card was restored to return program-specific results, which resolved this incident at 9:57 AM PT. ## Recurrence Prevention: As a recurrence prevention measure, we will be developing automated QA tests that will ensure all Insights reports return data that is specific to the program in which they are viewed and executed in, before any code changes are released to the production environment.

Read the full incident report →

Started Feb 10, 2026, 09:09 PM UTC· resolved Feb 19, 2026, 06:52 PM UTC

1. investigating Feb 10, 2026, 09:09 PM UTC

   We are currently investigating this issue.

2. identified Feb 10, 2026, 09:22 PM UTC

   AWS has identified an issue causing DNS resolution failures - more can be found here: https://health.aws.amazon.com/health/status

3. monitoring Feb 10, 2026, 10:20 PM UTC

   AWS has identified the root cause of the issue, implemented changes and can confirm significant recovery for the DNS resolution errors. There may still be some residual hiccups while the fixes propagate, however.

4. monitoring Feb 11, 2026, 02:58 PM UTC

   Amazon confirmed the issue has been fully resolved as of 8:18PM PT. Marking all components as fully operational.

5. resolved Feb 19, 2026, 06:52 PM UTC

   This incident has been resolved.

6. postmortem Feb 21, 2026, 01:53 AM UTC

   **Summary:** On February 10, 2026, between 20:47 UTC and 22:12 UTC, some customers experienced intermittent issues accessing Creator Studio. This was first observed internally and an incident was declared at 21:00 UTC. The issue was traced to a global DNS outage within Amazon Web Services \(AWS\). The outage caused inconsistent domain name resolution for certain services that rely on AWS DNS infrastructure. Because the disruption occurred at the cloud provider level, impact varied by geographic location. Once AWS resolved the underlying issue, service was fully restored. **Impact:** * Creator Studio was intermittently unavailable for some users depending on geographic location and DNS resolver behavior. * The issue was transient in nature but inconsistent; some users were able to access services while others could not, depending on how DNS responses were propagated in their region. * The disruption lasted approximately 1 hour and 25 minutes. **Root Cause:** AWS introduced a configuration change intended to improve its internal systems that inadvertently caused DNS records for some endpoints to resolve inconsistently across global DNS infrastructure. Because Firstup relies on AWS DNS and content delivery infrastructure, customers experienced intermittent domain resolution failures. The issue was external to Firstup’s infrastructure and could only be resolved by AWS.**Mitigation:** Upon internal detection of the issue at 20:47 UTC, the Cloud Operations and Engineering teams declared an incident. An investigation quickly confirmed the issue originated within AWS services, and we engaged our AWS Technical Account Manager. At 22:12 UTC, AWS confirmed the root cause of the issue was identified and significant recovery of the DNS propagation issue was seen. Firstup verified restoration through internal testing and confirmed access to affected services had returned to normal for all regions. AWS confirmed full resolution at 04:18 UTC. **Recurrence Prevention:** While this incident was caused by a cloud provider disruption outside of Firstup’s direct control, we are taking steps to improve detection and visibility: * Reviewing and repairing monitoring for Studio availability. * Expanding monitoring coverage to include additional geographic regions. * Auditing existing alert ownership to ensure monitoring issues are surfaced and resolved promptly.

Read the full incident report →

Started Dec 19, 2025, 02:32 PM UTC· resolved Dec 19, 2025, 10:50 PM UTC

1. investigating Dec 19, 2025, 02:32 PM UTC

   We are investigating a service disruption affecting email delivery on EU communities (US communities are not affected) This may be causing delays in email delivery from campaigns. Our next update will be in one hour.

2. investigating Dec 19, 2025, 03:37 PM UTC

   We are continuing to investigate this issue and are currently testing a potential solution. Campaigns are sending; however, delivery delays are still occurring.

3. monitoring Dec 19, 2025, 04:15 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 19, 2025, 10:50 PM UTC

   No further indications of email deliverability issues have been observed or reported. This incident is resolved.

5. postmortem Dec 30, 2025, 11:14 PM UTC

   **Summary:** On December 19 at 10:58 UTC, customers using campaign email delivery in the EU region experienced delays in email sending. Emails were successfully queued during the incident and delivered once the issue was resolved. An incident was declared after it was identified that campaign emails were not being sent as expected. The engineering team determined that an internal cleanup job caused unexpected load on the system, slowing email processing. A targeted hotfix was deployed to mitigate the issue, and normal operations fully resumed the same day, at 16:10 UTC. **Impact:** * Campaign emails for EU customers were significantly delayed. * No emails were lost; all queued emails were delivered after recovery. **Root Cause:** An internal cleanup job was run during normal peak hours, generating a large volume of backend operations. This coincided with high email traffic and triggered inefficient processing behavior, which delayed campaign email delivery across the EU region. **Mitigation:** Once the incident was declared, engineers identified an inefficient database query contributing to the slowdown. A targeted hotfix was deployed to bypass the slow operation, allowing email processing to recover. All queued emails were successfully delivered after the fix was applied. **Recurrence Prevention:** To minimize the likelihood of similar issues in the future, Firstup will take the following actions: * Move the EU email processing workload to a dedicated database cluster, aligning it with other regions. * Ensure internal cleanup jobs are restricted to off-peak hours. * Improve monitoring and alerting to provide earlier, high-signal detection of email processing delays.

Read the full incident report →

Started Dec 10, 2025, 05:49 PM UTC· resolved Dec 12, 2025, 06:47 PM UTC

1. investigating Dec 10, 2025, 05:49 PM UTC

   We are investigating reports where the selected branding color is showing up as red in the Employee Experience. We will provide you with an update within 1 hour.

2. identified Dec 10, 2025, 05:53 PM UTC

   We have identified a potential root cause for this issue and are currently testing a proposed fix. Another update will be provided within 1 hour.

3. identified Dec 10, 2025, 06:52 PM UTC

   The fix for this issue has been tested and is currently being deployed in the production environment. We will provide you with another update once it has been released, or within 1 hour.

4. monitoring Dec 10, 2025, 06:58 PM UTC

   The fix for this issue has now been released in the production environment. A browser refresh may be required to correct the impacted items. This incident is now resolved, and the impacted services will be placed under monitoring for now.

5. monitoring Dec 10, 2025, 07:17 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Dec 12, 2025, 06:47 PM UTC

   All impacted services have remained stable and fully functional. This incident is now resolved. A Root Cause Assessment will be published here once a postmortem of the incident has been completed.

7. postmortem Dec 17, 2025, 10:55 PM UTC

   **Summary:** On December 10, several customers observed that portions of their branded web experience appeared in red, instead of using the customer-selected accent color. An incident was declared after it was determined the issue affected multiple customers. The engineering team found that a prior internal update to a shared user-interface component had unintentionally removed color-related code that was still in use in production. When the web experience began using the updated version of this component after the December release cycle, this change caused the incorrect accent color to appear. A hotfix was issued the same day to restore the correct behavior and normal functionality was fully recovered for all customers approximately one hour after the incident was declared. **Impact:** * Affected elements included: * Sign-in and notification buttons * Branded blocks * Content cards * "Like" indicators * The translate option * No functionality was hindered; the issue was visual only. **Root Cause:** In early October, an update was made to an internal user-interface component. This update removed color-related code under the assumption that it was unused. However, parts of the web experience still depended on this code to correctly apply customer-selected accent colors. When the December release updated the web experience to use the latest version of this component, the unintended removal of that code introduced a visual regression. As a result, certain UI elements defaulted to red instead of the correct customer-specific color. **Mitigation:** Once the incident was declared, engineers quickly identified the regression and traced it to the internal component update. A targeted hotfix was prepared and deployed to restore correct color functionality across all affected elements. The incident was fully resolved approximately one hour after formal declaration. **Recurrence Prevention:** To minimize the likelihood of similar issues in the future, Firstup will take the following actions: * Strengthen regression testing when updating shared user-interface components, ensuring that changes are validated across all dependent surfaces.

Read the full incident report →

Started Nov 25, 2025, 05:49 PM UTC· resolved Nov 25, 2025, 10:46 PM UTC

1. investigating Nov 25, 2025, 05:49 PM UTC

   We are currently investigating an issue where some images requiring 3rd party transformation are not rendering as expected on Studio and Web Experience.

2. monitoring Nov 25, 2025, 06:28 PM UTC

   The issue affecting image transformations has been identified and mitigated, though there may be some residual effects related to caching. We're continuing to monitor the issue.

3. resolved Nov 25, 2025, 10:46 PM UTC

   We've been able to confirm the issue is mitigated, and no further reports have been made. Marking this incident resolved.

4. postmortem Dec 04, 2025, 12:05 AM UTC

   **Summary:** On November 25th, customers experienced issues where images that required transformation \(i.e., images with rounded corners, video preview images, inline images in rich text format\) stopped displaying across various parts of the platform. It became evident that the failure was related to a third-party image processing partner inadvertently disabling a critical feature used by Firstup during an automated service renewal process. All affected images began displaying normally once the configuration was fixed by the third-party provider. **Impact:** * Transformed images failed to display for customers. * No images were lost or corrupted; the issue solely affected their ability to be displayed. * Service returned to normal after the third-party provider corrected the configuration and cleared relevant caches. **Root Cause:** An incorrect change in service entitlements was made during the account renewal process with the provider. This misconfiguration impacted image delivery capabilities resulting in the “broken image” icon being displayed rather than the desired image. **Mitigation:** An incident was declared at 9:12 am PST. Upon clarifying the issue's scope and origin, engineers first validated that no recent Firstup code changes were responsible. Subsequent testing of image URLs and delivery paths led to the engagement of our third-party provider for deeper investigation. The provider confirmed the root cause was a misconfiguration in the account and their engineers applied a correction and cleared caches for the production environment, which subsequently restored image functionality. **Recurrence Prevention:** The following has been identified as follow-up to commit to as a part of this process: * Ensuring the necessary features are explicitly included in contract details during the contract negotiation process.

Read the full incident report →

Started Nov 14, 2025, 06:53 PM UTC· resolved Nov 15, 2025, 12:24 AM UTC

1. investigating Nov 14, 2025, 06:53 PM UTC

   We're currently investigating an issue where user imports become stuck in Created status.

2. identified Nov 14, 2025, 08:01 PM UTC

   The issue has been identified and work to mitigate any outstanding stuck imports is underway.

3. identified Nov 14, 2025, 08:21 PM UTC

   Imports are no longer in a 'stuck' state and the cause is understood. We're continuing to monitor to ensure a steady state.

4. monitoring Nov 14, 2025, 08:40 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Nov 15, 2025, 12:24 AM UTC

   No further issues have been noted since the fix was implemented; the incident is resolved.

6. postmortem Dec 02, 2025, 12:14 AM UTC

   **Summary:** On November 14, a large file uploaded by one customer program caused the system responsible for processing user imports to slow down significantly. Because the system processes import jobs in order, this slow-running file created a backlog that delayed other customers’ imports during the incident. Once the issue was identified, the problematic file was stopped and the system was restored, allowing imports to resume normal processing. **Impact:** * CSV files uploaded during the affected timeframe experienced delays. * No data loss occurred, and all delayed imports completed successfully once the system recovered. **Root Cause:** A large CSV file uploaded by one customer program contained a high volume of duplicate entries, which triggered recursive processing behavior. This caused the import system to spend an unusually long time processing that single file, slowing down the overall import queue. **Mitigation:** An incident was declared as soon as the scope was understood. After identifying the slow-running upload as the source of the delays, the team stopped the problematic job to prevent it from continuing to block the queue. Performance was then restored by restarting the processing workers and performing maintenance to improve lookup speeds. These steps allowed the system to clear the backlog and return to normal operation. All delayed import jobs completed successfully once the processing queue began moving again. **Recurrence Prevention:** The following actions have been identified as follow-up actions to commit to as a part of this process: * Adding monitoring that alerts us if import jobs begin taking unusually long or if the queue grows too quickly. * Improving how we validate uploaded user data. * Enhancing our import processing logic so that slow-running jobs can be isolated without delaying other customers.

Read the full incident report →

Started Oct 20, 2025, 01:18 PM UTC· resolved Oct 24, 2025, 11:42 PM UTC

1. investigating Oct 20, 2025, 01:18 PM UTC

   We are currently investigating an issue where Studio Insights Reports are not available on the Firstup US platform. Updates to follow asap.

2. investigating Oct 20, 2025, 01:37 PM UTC

   We are working with AWS to restore service to the Insights Reports. Initial investigation is pointing to a residual issue from the earlier Third Party issue with AWS.

3. investigating Oct 20, 2025, 02:26 PM UTC

   It has been confirmed that this is a residual issue from the AWS issue earlier and Engineering are working with AWS to resolve. It is not only Firstup impacted by this residual issue, Studio/ Web Experience and Mobile channels all remain available.

4. investigating Oct 20, 2025, 03:24 PM UTC

   AWS are still investigating the issue impacting Insights. There is also an issue impacting the display of some images on Web Experience and Campaigns sent, this is also third party caused and related to the AWS issue.

5. investigating Oct 20, 2025, 03:47 PM UTC

   We are continuing to investigate this issue.

6. investigating Oct 20, 2025, 04:56 PM UTC

   We are continuing to observe a low level of errors and intermittent network connectivity drops within the AWS cloud infrastructure. Service tuning on our side has been insufficient at times to handle the level of degredation we are observing with the underlying infrastructure. The following impact is being observed: * Web Experience - intermittent 502 Gateway Errors, blank white screen, errors loading content feed * Image transformation - some images are not displaying correctly and placeholders are being shown instead * User sync - we are reaching out directly to Customers we feel the user file upload may have failed while AWS S3 service wasn't available * Videos - AWS transcoding service appears to still be which is impacting video uploads

7. investigating Oct 20, 2025, 05:46 PM UTC

   We are continuing to observe a low level of errors and intermittent network connectivity drops within the AWS cloud infrastructure. Service tuning on our side has been insufficient at times to handle the level of degredation we are observing with the underlying infrastructure. The following impact is being observed: * Web Experience - intermittent 502 Gateway Errors, blank white screen, errors loading content feed * Image transformation - some images are not displaying correctly and placeholders are being shown instead * User sync - we are reaching out directly to Customers we feel the user file upload may have failed while AWS S3 service wasn't available * Videos - AWS transcoding service appears to still be which is impacting video uploads * Studio performance and availability degraded due to AWS backend network connectivity problems

8. monitoring Oct 20, 2025, 06:20 PM UTC

   We are observing substantial improvements in error rates platform wide. Moving incident to Monitoring status and Studio and Web Experience to fully operational. We have separate work streams for Insights, and web experience images intermittently not displaying.

9. monitoring Oct 20, 2025, 09:19 PM UTC

   We are continuing to monitor for any further issues.

10. monitoring Oct 20, 2025, 09:28 PM UTC

    Insights reporting on Firstup's US1 region remain unavailable while Amazon rebuilds the AWS Redshift data store that it relies on. We have received a number of progress updates from AWS throughout the day, but no firm commitment thus far of service restoration. All other Firstup services remain fully operational.

11. monitoring Oct 20, 2025, 09:50 PM UTC

    Insights reporting on the Firstup US1 region have now been restored and are fully operational. We will continue to keep all impacted services under monitoring for stability and functionality.

12. resolved Oct 24, 2025, 11:42 PM UTC

    AWS has remained stable and this incident is resolved.

13. postmortem Nov 03, 2025, 11:49 PM UTC

    ## Summary: On October 20th, 2025, at 12:05 AM PT, Firstup platform infrastructure monitoring tools fired multiple alerts of failed system health checks. Firstup engineers immediately acknowledged the alerts and started investigating. At 12:17 AM PT, the investigations revealed that both Studio and Employee Experience on the Firstup US platform were not accessible because of an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A platform incident was declared at 12:34 AM PT, and subsequently published on our [Status Page](https://status.firstup.io/incidents/26bv9gmp764g) at 12:44 AM PT. ## Severity: Sev-1 ## Scope: All customers on the Firstup US platform attempting to access Studio or the Employee Experience were in scope of this incident. ## Impact: Customers on the Firstup US platform primarily observed an error message while trying to log into or navigate within Studio or the Employee Experience. Some microservices on the Firstup US platform continued to exhibit degraded performance once access to Studio and Employee Experience was restored. These microservices included: · Insights Reports · Image/video uploading and processing · User Sync ## Root Cause: The root cause for this incident was determined to be an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A summary of the incident on the AWS platform can be found [here](https://aws.amazon.com/message/101925/). ## Mitigation: As AWS continued to work on the service disruption on their platform, access to Studio and Employee Experience on the Firstup US platform was restored by 2:37 AM PT, although some latency was still observed while navigating within these products. Some microservices on the Firstup US platform continued to experience intermittent degraded performance throughout the AWS incident, until it was fully resolved at 2:09 PM PT. ## Recurrence Prevention: In an effort to make the Firstup platform more resilient to an incident on a dependent third-party vendor such as AWS, we have adjusted our systems’ Health-checks and Liveliness probes to better react to pod slow startups which were observed to cause crash loops during the incident.

Read the full incident report →

Started Oct 20, 2025, 07:44 AM UTC· resolved Oct 20, 2025, 12:07 PM UTC

1. investigating Oct 20, 2025, 07:44 AM UTC

   Both Web Experience and Studio are currently down for US customers, EU Web Experience and Studio are available. The issue is linked to a broader AWS outage impacting US customers. We have reached out to AWS for updates.

2. investigating Oct 20, 2025, 07:57 AM UTC

   Update provided by AWS: We can confirm increased error rates and latencies for multiple AWS Services in the US. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause. We will provide an update in 45 minutes, or sooner if we have additional information to share.

3. identified Oct 20, 2025, 09:07 AM UTC

   Update from AWS Oct 20 2:01 AM PDT We have identified a potential root cause for error rates.... Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. We are working on multiple parallel paths to accelerate recovery.... We will continue to provide updates as we have more information to share, or by 2:45 AM.

4. identified Oct 20, 2025, 09:33 AM UTC

   Update provided by AWS: We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

5. identified Oct 20, 2025, 09:37 AM UTC

   Services are returning to normal after the latest update provided by AWS. Both Studio and Web Experience and all other channels are available. We are performing checkouts on all services. StatusPage will be set to Monitoring until AWS advises this issue has been fully resolved.

6. resolved Oct 20, 2025, 12:07 PM UTC

   Checkouts complete and no further reports from AWS in relation to any downtime expected. Root Cause Analysis will be published when provided by AWS.

7. postmortem Nov 03, 2025, 11:46 PM UTC

   ## Summary: On October 20th, 2025, at 12:05 AM PT, Firstup platform infrastructure monitoring tools fired multiple alerts of failed system health checks. Firstup engineers immediately acknowledged the alerts and started investigating. At 12:17 AM PT, the investigations revealed that both Studio and Employee Experience on the Firstup US platform were not accessible because of an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A platform incident was declared at 12:34 AM PT, and subsequently published on our Status Page at 12:44 AM PT. ## Severity: Sev-1 ## Scope: All customers on the Firstup US platform attempting to access Studio or the Employee Experience were in scope of this incident. ## Impact: Customers on the Firstup US platform primarily observed an error message while trying to log into or navigate within Studio or the Employee Experience. Some microservices on the Firstup US platform continued to exhibit degraded performance once access to Studio and Employee Experience was restored. These microservices included: · Insights Reports · Image/video uploading and processing · User Sync ## Root Cause: The root cause for this incident was determined to be an Amazon Web Services \(AWS\) service disruption in the N. Virginia \(us-east-1\) region. A summary of the incident on the AWS platform can be found [here](https://aws.amazon.com/message/101925/). ## Mitigation: As AWS continued to work on the service disruption on their platform, access to Studio and Employee Experience on the Firstup US platform was restored by 2:37 AM PT, although some latency was still observed while navigating within these products. Some microservices on the Firstup US platform continued to experience intermittent degraded performance throughout the AWS incident, until it was fully resolved at 2:09 PM PT. ## Recurrence Prevention: In an effort to make the Firstup platform more resilient to an incident on a dependent third-party vendor such as AWS, we have adjusted our systems’ Health-checks and Liveliness probes to better react to pod slow startups which were observed to cause crash loops during the incident.

Read the full incident report →

Started Oct 10, 2025, 12:27 PM UTC· resolved Oct 24, 2025, 11:45 PM UTC

1. investigating Oct 10, 2025, 12:27 PM UTC

   We are currently investigating a new platform incident impacting the web experience channel for some users. A 'oops something went wrong' error is being displayed on some programs. Updates to follow asap.

2. investigating Oct 10, 2025, 12:31 PM UTC

   EU customers are not impacted. Investigation is continuing.

3. investigating Oct 10, 2025, 12:58 PM UTC

   Work is continuing to fully restore service, improvements have been seen with Web Experience loading however it is still slow to load for some customers.

4. identified Oct 10, 2025, 01:22 PM UTC

   Web Experience is available and the load times have improved - we are continuing to monitor.

5. identified Oct 10, 2025, 01:28 PM UTC

   We are continuing to work on a fix for this issue.

6. monitoring Oct 10, 2025, 01:48 PM UTC

   A fix has been implemented and we are monitoring the results.

7. monitoring Oct 10, 2025, 04:48 PM UTC

   We continue to monitor for any further issues.

8. monitoring Oct 10, 2025, 05:39 PM UTC

   All services have been restored and are fully operational. We will continue to monitor the impacted services.

9. monitoring Oct 10, 2025, 05:40 PM UTC

   Residual slowness has been fully addressed and all components are now fully operational. Root cause ascertained, and impact has been fully mitigated. Risk of a recurrence is very low, but additional investigation is required for full resolution. This incident will remain in a monitoring status until that investigation and a recurrence prevention plan is committed, at which time an RCA (Root Cause Assessment) will be published.

10. investigating Oct 20, 2025, 04:37 PM UTC

    Moving back to phase I (unmitigated). Web Experience channel is frequently displaying a blank page followed by a 502 Gateway Error.

11. monitoring Oct 20, 2025, 04:38 PM UTC

    A fix has been implemented and we are monitoring the results.

12. resolved Oct 24, 2025, 11:45 PM UTC

    This incident is resolved.

13. postmortem Oct 24, 2025, 11:46 PM UTC

    **Summary:** On October 10th, 2025 we received alerts indicating there was a service disruption with the Web Experience channel and also customer reports of the following error being displayed on the Web Experience for impacted users: _‘Oops! Something went wrong.'_ This behavior was attributed to a change that was made to enable a verbose log setting on a back end system which generated an unexpected amount of load on the database for the US1 region. A verbose log setting provides detailed, step-by-step information to help with troubleshooting, debugging, and performance optimization by logging every action and event. More capacity was added to handle the additional load which restored service for the Web Experience however a residual issue remained with some slowness navigating on the Web Experience which was fully mitigated by disabling the verbose log setting. **Impact:** A system error message was returned to any end-user attempting to access/use the Web Experience between 05:00 AM and 05:56 AM PT \(56 minutes\). _‘Oops! Something went wrong.'_ Residual slowness was seen between 05:56 AM and 10:32 AM PT \(4 hours 26 minutes\). **Root Cause:** Root cause was determined to be due to a change made on October 9th to enable a verbose log setting on a back end system which generated an unexpected amount of load on the database for the US1 region, which, coupled with normal peaks in busyness, resulted in several services going into a restart loop. This restart loop ultimately could not be contained without causing a disruption in service to our Customers. The verbose log setting was enabled outside of business hours to assist with the troubleshooting and debugging of a prior incident on October 9th whereby Studio Insights Reports were not available for some users. This increase of the granular reporting logging events on database activity for audit/compliance unexpectedly generated a high level of load on the database during peak morning traffic US East coast hours. The frequent application restarts which caused the most significant amount of impact to users were the result of the database becoming too bogged down with writing the verbose log data creating I/O \(input/output\) boundness that blocked other services from interfacing with the database itself during peak traffic time. **Mitigation:** A platform incident was declared at 05:21 am PT on Friday, October 10th and the incident team restored service at 05:56 AM PT. Additional capacity was implemented to handle the unexpected load which mitigated the unavailability of the Web Experience channel. However residual slowness and latency were being experienced for some users until 10:32 AM PT until the verbose log setting was disabled. Database load then returned to normal levels. **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * A case was opened with AWS, our Cloud Provider to understand the nature of the load andto investigate how to optimize the logging without causing a platform performance degradation. This setting will not be re-enabled without full confidence that the issue will not reoccur. * Testing and Analysis - More rigorous testing analysis will be performed on UAT, if this setting is required to be enabled again, to detect any errors/irregularities before it is enabled in Production. * Improve real-time monitoring for services going into a restart loop and also average latency times so an unexpected spike can be detected faster.

Read the full incident report →

Started Oct 09, 2025, 05:29 PM UTC· resolved Oct 13, 2025, 03:58 PM UTC

1. investigating Oct 09, 2025, 05:29 PM UTC

   We are currently investigating an issue where Studio Insights Reports are not available on the Firstup US platform. We will provide you with an update within 1 hour.

2. monitoring Oct 09, 2025, 06:05 PM UTC

   We have mitigated this incident, and Insights Reports on the Firstup US platform are now available. A screen refresh may be required to load the reports. We will be placing the impacted components under monitoring for now.

3. resolved Oct 13, 2025, 03:58 PM UTC

   This incident is now resolved, and all impacted services have remained available and fully operational. A Root Cause Assessment will be provided here once a postmortem review of this incident is concluded.

4. postmortem Oct 22, 2025, 01:48 PM UTC

   ## Summary: On October 9th, 2025, starting at around 10:03 AM PT, we received reports where Studio users received a “You don’t have access to Insights Reports” error notification while trying to access Studio Insights Reports. A platform incident was declared at 10:23 AM PT and published on our Status Page at 10:29 AM PT, as the incident response team investigated the issue. ## Severity: Sev-2 ## Scope: All Studio users for customers on the US Firstup platform attempting to access Studio Insights Reports were in scope of this incident. ## Impact: Any Studio user on the US Firstup platform observed a “You don’t have access to Insights Reports” error notification while trying to access Studio Insights Reports in the duration of this incident \(59 minutes\), rendering the reports unavailable to the user. Customers on the EU Firstup platform were not impacted by this incident. ## Root Cause: The root cause for this incident was determined to be user error when a Firstup engineer accidentally added the program IDs for customers on the US Firstup platform to the Insights Reports blocked list \(which takes a priority over the allowed list\) in the production database. This was a manual process and a cut and paste error. ## Mitigation: To mitigate this incident, the latest production database snapshot was restored, and the Insights Reports microservice was repointed to the now-good snapshot at 11:02 AM PT, restoring Insights Reports access to Studio users. ## Recurrence Prevention: To prevent this incident from happening again, we have implemented a peer review process for any blocked programs additions and subtractions as part of our Insights Reports production configuration change management.

Read the full incident report →

Started Sep 17, 2025, 07:00 PM UTC· resolved Sep 24, 2025, 07:38 PM UTC

1. investigating Sep 17, 2025, 07:00 PM UTC

   We are investigating reports where some campaign creators are receiving an error message when adding a link to a link block. We will provide an update within 1 hour.

2. identified Sep 17, 2025, 07:48 PM UTC

   We have identified a potential cause for this issue and are currently working to mitigate it. Another update will be provided within 1 hour.

3. identified Sep 17, 2025, 08:45 PM UTC

   We continue to work on a fix to mitigate this issue. Another update will be provided within 1 hour.

4. identified Sep 17, 2025, 09:40 PM UTC

   We continue to work on a fix to mitigate this issue. Another update will be provided within 1 hour.

5. identified Sep 17, 2025, 10:37 PM UTC

   A mitigating fix for this issue has been developed and is currently undergoing QA testing. Another update will be provided within 1 hour, or as soon as the fix has been released.

6. identified Sep 17, 2025, 11:37 PM UTC

   The proposed fix has passed QA testing, and we are working on deploying it in the production environment. Another update will be provided within 1 hour, or as soon as the fix has been released.

7. monitoring Sep 17, 2025, 11:56 PM UTC

   The proposed fix has been released. We will be placing the impacted services under monitoring for now.

8. resolved Sep 24, 2025, 07:38 PM UTC

   This incident is now resolved, and the impacted service has remained available.

9. postmortem Oct 16, 2025, 10:43 PM UTC

   **Summary:** On September 17th, 2025, at 5:56 AM PT, we received a single report where a Studio user received an error message on the Campaign Design page when adding a specific link to a link type block, impairing the full use of the link type block. Initial troubleshooting was able to reproduce this issue, and an internal platform incident was declared at 9:06 AM PT. Once the incident response team determined that the scope of this issue was platform-wide, it was published on our Status Page at 12:00 PM PT. **Severity:** Sev-2 **Scope:** Some Studio users could not add specific links to the link type block in the Campaign Design page. **Impact:** Some Studio users received an error message on the Campaign Design page when adding specific links to a link type block, impairing the full use of the link type block in the duration of this incident \(22hr 23mins\). A refresh of the page restored its functionality, but any attempt to add the same “problematic” link resulted in the same error displayed. **Root Cause:** The root cause for this incident was attributed to a code change that was deployed on September 16th, 2025 at 6:33 PM PT. The new code did not account for fields \(title, description, body, etc.\) that returned a “null” value after the entered link was scraped, resulting in the code to error. This incident did not impact all Studio users, as links that did not return fields with a “null” value were being processed as expected. **Mitigation:** To mitigate this incident, the new code was rewritten to account for fields that returned a “null” value and redeployed at 4:56 PM PT after passing QA tests. **Recurrence Prevention:** To prevent this incident from happening again, we have added this edge case to our QA test plan before any code-change release to the production environment.

Read the full incident report →

Started Aug 15, 2025, 02:46 PM UTC· resolved Aug 16, 2025, 07:59 AM UTC

1. investigating Aug 15, 2025, 02:46 PM UTC

   We are investigating an issue affecting User Sync (User File via SFTP processing). This is resulting in user sync jobs encountering elevated error rates. You may see the user sync email showing that many records were not processed due to errors. Our next update will be in 30 minutes.

2. monitoring Aug 15, 2025, 03:22 PM UTC

   We have applied a fix for the issue affecting user sync processing. We are reprocessing errored jobs and closely monitoring the results.

3. resolved Aug 16, 2025, 07:59 AM UTC

   This incident has been resolved. All previously errored user sync files have now been successfully reprocessed, and summary emails were sent as expected. We have been monitoring the user sync service overnight and no further issues have been observed. Any new errors that appear will be valid user sync processing errors and can be addressed using the guidance in our Help Centre article: https://support.firstup.io/hc/en-us/articles/13072635044503-SFTP-File-Upload-Errors-and-User-Sync-Results If you have any questions or encounter further issues, please reach out to our Support team at support.firstup.io.

4. postmortem Sep 11, 2025, 11:53 PM UTC

   **Summary**: On August 14th, 2025, at approximately 8:00 PM PT, we began experiencing elevated error rates in user sync jobs across several customer programs. The issue was due to a change introduced during a recent hotfix, which impacted how user identifier matching was processed. As a result, many user sync jobs failed with higher-than-normal error rates. A platform incident was declared on August 15th, and our engineering teams immediately worked to restore normal service. By the evening of August 15th, all impacted jobs had been successfully reprocessed, and user sync services were fully restored. **Impact**: 1\. Users in affected programs were not updated correctly during the incident window 2\. More than 260 sync jobs required reprocessing. 3\. Data inconsistencies persisted until the reprocessing was complete. **Root Cause**: A hotfix deployed on August 14th introduced a defect in user sync processing logic. Specifically, the changes altered the handling of user identifier matching resulting in widespread “two different users found” errors. **Mitigation**: 1\. The defective hotfix was rolled back. 2\. Stuck jobs were cleared to allow downstream jobs to process. 3\. All impacted jobs were reprocessed to ensure data consistency. 4\. Additional monitoring was added to identify elevated error rates more quickly in the future. **Recurrence Prevention**: The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: 1\. Adjusting monitoring thresholds to catch sync job error spikes earlier. 2\. Improving detection of long-running or blocked jobs. 3\. Enhancing hotfix testing to better cover user sync workflows. 4\. Strengthening post-hotfix monitoring to validate stability and prevent recurrence.

Read the full incident report →

Started Aug 05, 2025, 03:44 PM UTC· resolved Aug 26, 2025, 12:32 PM UTC

1. investigating Aug 05, 2025, 03:44 PM UTC

   We are currently investigating a platform incident impacting the web experience channel for some users. A 'You've been signed out or your access has been expired' is appearing on some communities. Updates to follow asap.

2. investigating Aug 05, 2025, 03:53 PM UTC

   Cause of issue has been identified and we are currently taking action to mitigate which will resolve the authentication errors being displayed.

3. monitoring Aug 05, 2025, 04:05 PM UTC

   Immediate impact has been mitigated. Errors are no longer appearing and successful log in checks have been done. Stability has been restored and we are continuing to monitor. Root cause to follow.

4. monitoring Aug 05, 2025, 04:05 PM UTC

   We are continuing to monitor for any further issues.

5. investigating Aug 05, 2025, 06:37 PM UTC

   We have seen a recurrence of this issue and are currently investigating. Same authentication errors are being displayed for some programs. Updates to follow asap.

6. investigating Aug 05, 2025, 06:39 PM UTC

   Moving incident back to active state. Mitigation efforts have been insufficient to fully isolate Customers from impact. Symptoms remain similar where users are being signed out, but other connection errors may be intermittently displayed as well.

7. identified Aug 05, 2025, 07:18 PM UTC

   Root cause is identified and we are actively working to mitigate the issue.

8. monitoring Aug 05, 2025, 07:27 PM UTC

   Impact has now been fully mitigated and we do not expect a recurrence. This incident will remain in monitoring state while we evaluate the longer term mechanism required to completely resolve the issue.

9. resolved Aug 26, 2025, 12:32 PM UTC

   This incident has been resolved.

10. postmortem Aug 26, 2025, 12:32 PM UTC

    **Summary:** On August 5th, 2025 we began receiving Customer reports of the following error being displayed by the Web Experience channel for impacted users: _"You've been signed out or your access has been expired"._ This behavior was attributed to unexpected system memory pressure which resulted in resource starvation on the platform and subsequent intermittent user session invalidation. The initial attempt to mitigate impact by doubling the size of the system cache worked temporarily. However, further action was needed to quarantine a recently published campaign determined to be the cause of the out-of-memory condition. This secondary action fully restored service for the Web Experience channel. ‌ **Impact:** A system error message was displayed intermittently to end-users accessing the Web Experience between 08:21 AM - 09:01 AM PT and again between 11:23 AM - 11:37 AM PT. _"'You've been signed out or your access has been expired"_ Other services and endpoints including Creator Studio and the Mobile Experience remained fully functional. ‌ ​**Root Cause:** Root cause was determined to be data store instances being above memory capacity resulting in intermittent user session invalidation. This behavior was attributed to unexpected increased memory utilization due to several factors: 1. Large number of communications/campaigns being sent at the same time near US peak hours which increased cache-pressure. 2. Non-optimal caching of a recently published Classic Studio campaign containing multiple large images in the campaign content body itself. 3. Increase in ‘401 unauthorized’ errors due to a newly discovered software bug which directly led to the _"You've been signed out or your access has been expired"_ error message on the front end. ‌ **Mitigation:** A platform incident was declared at 08:31 am PT on Tuesday, August 5th and at 08:41 AM PT the incident team determined the root cause to be related to memory exhaustion. Issue was attempted to be mitigated earlier by and faster by doubling the size of the cache\(data store\) instance which temporarily restored service. The incident recurred approximately 2 hours later, and further analysis revealed non-optimal storage-reuse \(ie caching\) behavior related to a recently published campaign. That implicated campaign was quarantined to address the immediate impact, thus fully restoring service for the Web Experience channel. ‌ **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * Campaign body caching has been optimized so we will only cache the data that is being used in the response. This change has already been completed, and we have observed significant improvement in cache optimization by both reducing the size of allowed cacheable objects and also caching the same object for multiple users. * Additional real-time monitoring has been implemented to improve threshold warnings for high memory usage. * We have implemented a hotfix for the behavior causing the user session invalidation error message so that all cache-calls go through new logic that fall back to the source of truth and are resilient to Data Store cache issues. This has put us in a state where we will not have a re-occurrence of the unintended user-facing error message, and will drastically reduce the chance of using up the full memory. * Testing and Analysis - Rigorous load testing on non-production internal environments was performed to verify expected behavior for campaigns with large audiences containing imagines inline with the content body.

Read the full incident report →

Started Aug 04, 2025, 06:59 PM UTC· resolved Aug 18, 2025, 04:47 AM UTC

1. investigating Aug 04, 2025, 06:59 PM UTC

   We are currently investigating a new platform incident impacting the web experience channel for some users.

2. investigating Aug 04, 2025, 07:15 PM UTC

   Investigation is continuing in to possible root cause. An incident bridge has been initiated and primary focus is on the source of the error message currently being displayed for impacted users: "Error: Minified React error #300; visit https://reactjs.org/docs/error-decoder.html?invariant=300 for the full message or use the non-minified dev environment for full errors and additional helpful warnings." A change rollback is underway for the most recent change based on time correlation.

3. investigating Aug 04, 2025, 07:35 PM UTC

   Rolling back most recent change has not fixed the underlying problem. Focus shifting to local browser cache management given the impact is intermittent. No ETA yet for full restoration of service. Scope remains limited to the Web Experience. Other service endpoints remain fully operational.

4. identified Aug 04, 2025, 08:02 PM UTC

   Immediate impact has been mitigated. Work continues to determine root cause. The Custom Homepage Degradation feature has been disabled in the interim. A follow-up communication will be provided once additional information is available and regularly timed updates will cease so long as the impact remains mitigated.

5. monitoring Aug 04, 2025, 08:18 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Aug 18, 2025, 04:47 AM UTC

   This incident has been resolved.

7. postmortem Aug 18, 2025, 04:47 AM UTC

   **Summary:** On August 4th, 2025 we began receiving Customer reports of the following error being displayed on the Web Experience for impacted users: "_Error: Minified React error #300; visit_ [_https://reactjs.org/docs/error-decoder.html?invariant=300_](https://reactjs.org/docs/error-decoder.html?invariant=300) _for the full message or use the non-minified dev environment for full errors and additional helpful warnings._" This behavior was attributed to a change that was made as a part of the hot fix deployed earlier in the day that adjusted the retention period for local \(web browser\) storage of specific files used to render the Web Experience. An unexpected side effect from that change resulted in content attempting to be rendered prematurely--and the low level react error being displayed to all users during the time the platform incident was active. The initial attempt to rollback the hot fix was insufficient, forcing a more invasive removal of the entire local storage management functionality This action fully restored service for the Web Experience channel. **Impact:** A system error message was returned to any end-user attempting to refresh the Web Experience between 11:51 AM and 13:03 PM PT \(1 hour, 24 minutes\). "Error: Minified React error #300; visit [https://reactjs.org/docs/error-decoder.html?invariant=300](https://reactjs.org/docs/error-decoder.html?invariant=300) for the full message or use the non-minified dev environment for full errors and additional helpful warnings." Other services and endpoints including Creator Studio and the Mobile Experience remained fully functional. **Root Cause:** Root cause was determined to be due to a software change made earlier in the day on August 4th as a part of a hot fix which altered the retention period for local \(web browser\) storage of specific files used to render the Web Experience. A few months ago, we completely changed the behavior for how the Firstup Web Experience loads to make it more resilient to backend failures, planned maintenance activities, and hiccups in client network connectivity. Short of a full offline mode, focus was made around custom homepage content, quicklinks, and other features and functionality that comprise the Firstup Employee Intranet offering. This work changed how the application itself loads--at a high level it fires up with all of the content it has stored locally, and then checks to see if it can refresh the content \(if it is new, accessible to the client…etc\). The hot fix itself that triggered this platform incident unexpectedly broke how the Web Experience application initialization worked that is used by this new local caching mechanism, such that the app attempted to render content before everything it needed to render it successfully had been loaded from the local web browser storage \(cache\)--resulting in the low level react error from the underlying application framework. **Mitigation:** A platform incident was declared at 11:51 am PT on Monday, August 4th and the incident team determined the root cause at 12:10 pm PT. Issue was attempted to be mitigated earlier and faster by rolling back the hot fix which proved to be insufficient, so a decision was made to remove the entire local storage management functionality that had been added several months back to improve the graceful degradation of Custom Homepage, Quicklinks, and other products that comprise the Firstup Employee Intranet offering. By not utilizing local storage \(caching\) at all, the early attempt to render content before the app was ready was corrected indirectly, thus mitigating the impact to the Web Experience channel. **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * Testing and Analysis - More rigorous testing on UAT for client side changes and analysis to detect any errors/irregularities before a change goes live. * Additional real-time monitoring to be implemented that will alert the Engineering team of client-side only errors \(on authenticated pages, such as the Web Experience homepage\). * Improvement has been identified for the rollback scripts which will reduce rollback time to seconds in the event of an incident. * Add mechanism to clear client side cache more easily without the need for a platform wide emergency change for cases such as this.

Read the full incident report →

Started Jul 23, 2025, 06:32 PM UTC· resolved Jul 28, 2025, 03:55 PM UTC

1. investigating Jul 23, 2025, 06:32 PM UTC

   We are investigating reports where the direct delivery and audiences page is stuck in a loading loop. We will provide an update within 1 hour.

2. identified Jul 23, 2025, 07:30 PM UTC

   We have identified a potential root cause for this issue, and are working to mitigate it. Another update within 1 hour.

3. identified Jul 23, 2025, 08:49 PM UTC

   We continue to work on mitigating this issue. Another update within 1 hour.

4. monitoring Jul 23, 2025, 10:33 PM UTC

   This issue has been mitigated, and the impacted services should now be fully operational. We will be placing these services under monitoring for now.

5. resolved Jul 28, 2025, 03:55 PM UTC

   This incident is now resolved, and all associated services have remained available and fully operational.

6. postmortem Aug 18, 2025, 08:54 AM UTC

   **Summary:** On July 23rd, 2025 we began receiving Customer reports of Audience and Direct Delivery pages failing to load. Audience pages were either returning HTTP 500 errors hanging during the page loading. This behavior was attributed to a code change which was released on July 22nd to improve detection of circular audience references. Circular audience references can occur when one audience refers to itself by being included in another existing audience. This code change had an unintended impact on page loading performance and audience resolution and was subsequently rolled back. **Impact:** Performance degradation was observed across multiple areas of the Creator Studio, resulting in blocking some users from publishing Direct Delivery campaigns. The following symptoms were observed while the problem was present: 1. The Direct Delivery page failed to load or timed out for users across programs. 2. Audience pages frequently returned 500 errors or hung during initial loading. ‌ **Root Cause:** Root cause was determined to be due to a code change which was released on July 22nd that improved detection of circular audience references. Circular audience references when recalculated may result in errors or the audience itself being put into an erroneous state which will appear as archived and not able to be used in new or existing campaigns. The change made to prevent this condition from occurring had an unintended impact which led to drop in caching hit rates for the audience component, causing performance degradation in any Studio features relying on audience resolution. A cache hit allows the audience compiler to reuse a stored result instead of recompiling the audience being referenced from scratch. ‌ **Mitigation:** A platform incident was declared at 11:20 am PT on Wednesday, July 23rd and the incident team determined the root cause at 11:56 pm PT. Issue was mitigated by rolling back that change and the cache hit rate returned to >98% from around 30%. Rollback completed by 3:30 pm PT mitigating the negative page performance issues seen on the Audience and Direct Delivery pages. ‌ **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: ● Load Testing and Analysis - More rigorous load testing and analysis to detect performance issues or latency spikes before a change goes live. ● Additional real-time monitoring has been implemented that will alert the Engineering team of overall % change in audience query compilation cache hit rate. ● Enhance our software change management policies to use feature flags for rollout of changes in performance sensitive areas of the platform to reduce rollback time.

Read the full incident report →

Started Jul 08, 2025, 03:49 PM UTC· resolved Aug 06, 2025, 06:10 PM UTC

1. investigating Jul 08, 2025, 03:49 PM UTC

   We have received several reports where Email Campaign metrics are inaccurate and showing over 100% open rates since June 3rd, 2025. We are currently investigating this issue and will provide an update within 1 hour.

2. investigating Jul 08, 2025, 04:45 PM UTC

   We continue to investigate this issue and will provide another update within 1 hour. The current working hypothesis includes underreported email deliveries, resulting in reports that may be showing over 100% email open rates, as well as 0% delivery and open rates in some reports, even though the emails have been delivered successfully.

3. investigating Jul 08, 2025, 05:45 PM UTC

   We continue to investigate this issue. Another update within 1 hour.

4. identified Jul 08, 2025, 06:18 PM UTC

   We have identified a potential root cause of this issue and are working to mitigate the problem. Another update within 1 hour.

5. monitoring Jul 08, 2025, 06:42 PM UTC

   This issue has now been mitigated. Email campaigns published after 11:15 AM PT today should now show accurate email delivery and open rates (when available). We will be placing the impacted services under monitoring for now. We will also be exploring options on whether it is possible to correct the impacted reports between July 3rd, 2025, through July 8th, 2025, at 11:15 AM PT.

6. resolved Aug 06, 2025, 06:10 PM UTC

   This incident has now been fully resolved and all products and components affected remain fully operational.

7. postmortem Aug 06, 2025, 06:11 PM UTC

   **Summary:** On July 8th, 2025 we began receiving Customer reports of email campaign metrics showing unusual numbers including engagement %ages greater than 100%, deliveries well below expected levels, and unique click counts higher than open rates. This behavior was attributed to a standardized Amazon Web Services \(AWS\) inbound application firewall rule added as a part of a security initiative to continually improve the overall security posture of the Firstup platform on an ongoing basis. Most missing data was able to be recovered and backfilled into respective Customer reports with an average of around 1% standard deviation of remaining missing events from July 2nd through July 7th. **Impact:** Scope of the issue was limited to email campaign delivery metrics and any reports or data exports dependent on email engagement data. While the platform incident remained active, and before data backfill efforts concluded, email metrics including delivery count, engagement percentage, number of unique opens, and number of unique clicks, remained inconsistent.Engagement Percentage in particular showed elevated numbers because delivery events may not have been fully recorded, while subsequent opens or click events for the same message ID were correctly recorded, allowing for %ages in excess of 100%, because the Insights calculations operate on the data available to them at the time. **Root Cause:** Root cause was determined to be due to the addition of a standardized AWS-recommended default application firewall rule intended to drop inbound Application Programming Interface \(API\) traffic with unusually large payload sizes. This new rule, SizeRestrictions\_BODY is outlined in the following document as a part of Amazon’s Baseline rule groups: [https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.htmlThis](https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.htmlThis) additional rule negatively impacted webhook data containing user email engagement events sent from Firstup’s 3rd party email provider as those requests frequently batched data together for more efficient delivery–but caused the payload size to exceed what was allowed by the new rule \(8KB\), consequently dropping the traffic as it was assumed to be malicious in nature.The timing of the change prior to the US July 4th holiday, where total email volume was much lower, contributed to an overall delayed discovery of the underlying problem until after a full day of normal campaign volumes resumed the following Monday, July 7th. **Mitigation:** A platform incident was declared at 10:40am PT on Tuesday, July 8th and the incident team determined the root cause several hours later at 1:15pm PT. The offending firewall rule was disabled 15 minutes later, effectively mitigating the condition from persisting any further.All webhook data for events that occurred on July 8th were reprocessed automatically within 24 hours, effectively narrowing the remaining impact window to only July 2nd through July 7th. Because the data was dropped at the firewall, Firstup reached out to the third party email provider to see if there was a way to export the missing data presumed to be still stored on their systems. This initial data extraction took several days to complete and a program to ingest it safely and update the historic reporting data was developed concurrently.Over the course of the next several days and weeks, there were several iterations of backfill as missing events were found that required further extractions and updates making the backfill process more complicated and prolonged than it would typically be.On Monday, August 4th, Firstup’s third party email provider informed us that any remaining missing events–primarily outstanding delivery events, would not be recoverable due to a software bug on their own systems. As a result, all available backfill data is complete, and it is estimated that the standard deviation of irrecoverable events is 1% across the full duration of the platform incident. **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * Change management Standard Operating Procedures to add new rules in a warning/count mode only instead of block mode unless explicitly countering a known and active threat. * Additional real-time monitoring that will alert the Cloud Operations team of overall % change in detections/blocks in the WAF from the baseline levels. * Continued service segmentation at the API gateway so that impact to specific business processes or production data are easier and faster to isolate.

Read the full incident report →

Started May 15, 2025, 01:58 PM UTC· resolved May 23, 2025, 04:33 PM UTC

1. investigating May 15, 2025, 01:58 PM UTC

   We are investigating a service disruption affecting the Employee Experience and Shortcuts. The Employee Experience (web and mobile) may be intermittently unavailable for some users, and we have reports of Shortcuts missing within the experience once accessed. Our next update will be in 30 minutes.

2. investigating May 15, 2025, 02:39 PM UTC

   We are continuing to investigate the service disruption affecting the Employee Experience and Shortcuts. Our next update will be in 30 minutes.

3. monitoring May 15, 2025, 03:11 PM UTC

   We have applied a fix for the issue affecting Employee Experience and Shortcuts and are currently monitoring the platform.

4. resolved May 23, 2025, 04:33 PM UTC

   This incident is now resolved, and all impacted services have remained stable and available during the monitoring phase. A Root Cause Assessment will be published here as soon as a postmortem of the incident is completed.

5. postmortem Jun 04, 2025, 08:47 PM UTC

   ## Summary: On May 15th, 2025, starting at around 6:12 AM PT, we received reports where some end users on the Firstup platform were either unable to load the Employee Experience, experienced slow loading, or observed that some shortcuts were missing with an error message displayed above the available shortcuts. Upon initial investigations, it was determined that the issue was reproducible but intermittent. A platform incident was declared at 6:49 AM PT and published on our Status Page at 6:58 AM PT, and the incident response team continued with the investigations. ## Severity: Sev-2 ## Scope: Some end users for customers on the Firstup platform attempting to access or navigate within the Employee Experience. ## Impact: Some end users on the Firstup platform were either intermittently unable to load the Employee Experience, experienced slow loading, or observed that some shortcuts were missing with an error message displayed above the available shortcuts in the duration of this incident \(1hr 53mins\). ## Root Cause: The root cause for this incident was determined to be an Out-Of-Memory \(OOM\) condition in the back-end service which provides the primary API that the web and mobile experience applications communicate with. This OOM condition was attributed to a memory leak in the version of the programming language this service is written in, resulting in the service’s inability to fulfill requests from the Employee Experience intermittently. ## Mitigation: To mitigate this incident, the incident response team performed several actions to restore services, including restarting the back-end service to free up memory, temporarily increased the memory size, and scaled up the number of processing pods \(in effect replacing pods which were stuck in a CrashLoopBackOff state because they were unable to communicate with the impacted service\) to facilitate requests from the Employee Experience during the peak traffic time. ## Recurrence Prevention: To prevent this incident from happening again, we have upgraded to a more recent version of the programming library that had been attributed as the cause of the original memory leak.

Read the full incident report →

Started May 06, 2025, 09:52 AM UTC· resolved May 09, 2025, 05:26 PM UTC

1. investigating May 06, 2025, 09:52 AM UTC

   We are urgently investigating an issue with intermittent unavailability of the platform being experienced for both Studio and Web Experience. EU customers are impacted. Updates to follow asap.

2. investigating May 06, 2025, 11:15 AM UTC

   We have identified the cause of this issue and steps are being taken to mitigate, we are also seeing an improvement with performance and both Studio/Web Experience are loading correctly. Issue will be fully resolved in 10 to 15 minutes while final mitigation steps are taken.

3. monitoring May 06, 2025, 11:33 AM UTC

   All mitigation steps have been been taken to resolve and stability has been restored. Both Studio/Member Experience are available. Full details of root cause to follow.

4. resolved May 09, 2025, 05:26 PM UTC

   This incident is now resolved, and all impacted systems have remained stable and fully available. A postmortem for this incident will be published here as soon as it becomes available.

5. postmortem May 13, 2025, 04:56 PM UTC

   ## Summary: On May 6th, 2025, at 9:16 AM UTC, we received a report where some users on the Firstup EU platform were unable to load the Studio UI. Upon initial investigations, it was determined that the issue was reproducible but intermittent. A platform incident was thus declared at 9:25 AM UTC, and an incident response team continued with the investigations. ## Severity: Sev-2 ## Scope: Only customers on the Firstup EU platform were in scope of this incident. ## Impact: Some Studio users on the Firstup EU platform were unable to successfully load their Studio instance and observed an infinite loading animation during this incident \(2hrs 59mins\). A page refresh intermittently allowed some users’ Studio instance to load successfully. Users on the Firstup US platform were not impacted by this incident. ## Root Cause: The root cause for this incident was determined to be an automation test that started running on the Firstup EU production platform at 8:30 AM UTC, which appeared to create a significant increased load on a backend system that serviced Studio. The increased load elevated the latency on the system and resulted in the system being unable to respond to Studio requests in a timely manner. ## Mitigation: The system self-recovered and Studio users started observing Studio loading successfully, as the latency reduced with the automation test job nearing its completion, which fully completed running at around 11:29 AM UTC. We also temporarily halted any other automation testing on the production environment. ## Recurrence Prevention: To prevent this incident from happening again, we will: * Reduce the frequency and quantity of automation tests on the production environment. * Improve the stability of tests in the testing environment, to reduce reliance on the production environment for tests.

Read the full incident report →

Started Apr 23, 2025, 09:00 PM UTC· resolved Apr 23, 2025, 09:00 PM UTC

1. resolved Apr 23, 2025, 10:42 PM UTC

   Between 2 and 3 PM PT some emails may have been delayed due to a third party related incident detailed here: https://status.sendgrid.com/incidents/6fj45kvpw8n3 . We are continuing to actively monitor the situation and working with the third party to understand root cause and scope. Monitoring has indicated no further delays, but there may be a chance that email addresses previously on a bounce suppression list will receive emails if they continue to be included in active audiences.

Read the full incident report →

Started Mar 26, 2025, 04:11 PM UTC· resolved Mar 31, 2025, 03:29 PM UTC

1. investigating Mar 26, 2025, 04:11 PM UTC

   We are currently investigating reports where some users are receiving an error message while trying to log into the Employee Experience. We will provide you with an update in 1 hour.

2. monitoring Mar 26, 2025, 04:18 PM UTC

   The Employee Experience is now fully restored and accessible. We will be placing the impacted services under monitoring for now.

3. resolved Mar 31, 2025, 03:29 PM UTC

   This service degradation is now resolved, and all associated systems have remained stable and fully available.

4. postmortem Apr 14, 2025, 05:27 PM UTC

   ## Summary: On March 26th, 2025, at 8:48 AM PT, system monitors alerted us of a potential issue with connecting to our database, and our incident response team immediately started looking into the alert. Starting at around 8:55 AM PT, we started receiving customer reports where users saw an error message while trying load the Employee Experience. Following initial correlation of related events, a platform incident was declared at 9:04 AM PT, and published on our Status Page at 9:11 AM PT. ## Severity: Sev-2 ## Scope: Any end-user attempting to access or navigate through the Web or Mobile Employee Experience. ## Impact: A system error message was returned to any end-user attempting to access or navigate through the Web or Mobile Employee Experience between 8:48 AM and 8:58 AM PT \(10 minutes\). ## Root Cause: The root cause was determined to be an exhaustion of available connection to our database. ## Mitigation: The system self-healed by 8:58 AM PT as various services associated with the Employee Experience auto-restarted to release any unused/stale connections to the database, in effect reducing the total number of connections to the database, thereby allowing end-user requests from the Employee Experience to establish new connections with the database. ## Recurrence Prevention: The following system enhancements have been implemented or have been identified as a follow-up item to prevent a recurrence of this incident due to the same root cause: * We have performed database connection tuning by renaming the connections to utilize the _service\_name_ that serves multiple applications, as opposed to utilizing the individual _application\_name_. This has encouraged connections re-use by applications within a specific service, and has lowered the overall number of connections to the database by over 50%. * We are looking to upgrade the Postgres Ruby library to better handle connection pinning in the database, which will facilitate more connection re-use. This is slated to be completed in our Scheduled Software Release window on 4/15/2025.

Read the full incident report →