Firstup incident

Web Experience and API are currently unavailable for EU Customers.

Notice Resolved View vendor source →

Firstup experienced a notice incident on May 31, 2024 affecting EU Firstup Platform and Partner API and 1 more component, lasting 4d 14h. The incident has been resolved; the full update timeline is below.

Started
May 31, 2024, 08:42 AM UTC
Resolved
Jun 04, 2024, 10:56 PM UTC
Duration
4d 14h
Detected by Pingoru
May 31, 2024, 08:42 AM UTC

Affected components

EU Firstup PlatformPartner APIWeb Experience

Update timeline

  1. investigating May 31, 2024, 08:42 AM UTC

    We are investigating a service disruption affecting Web Experience for EU customers. Our Engineering Team are urgently investigating. Updates to follow.

  2. identified May 31, 2024, 09:09 AM UTC

    We have identified a possible root cause for the issue affecting Web Experience and the Partner API for EU customers and will provide further updates once we have applied a fix.

  3. monitoring May 31, 2024, 09:39 AM UTC

    We have applied a fix for the issue affecting Web Experience and Partner API for EU communities and are currently monitoring.

  4. resolved Jun 04, 2024, 10:56 PM UTC

    This incident has been resolved.

  5. postmortem Jun 10, 2024, 03:10 PM UTC

    **Summary:** On Thursday, May 31st, 2024, we received reports from several EU communities that the Web and Mobile Experiences were malfunctioning. End users reported that the Web or Mobile Experiences weren’t loading as expected, and a blank screen was observed. **Scope:** The scope of this incident was restricted to end users trying to access the Web and Mobile Experiences for communities in the EU only. **Root Cause:** A platform incident was declared at 08:30 UTC, and the incident response team identified a misconfiguration on the EU instance of CloudFront cache policy. The misconfiguration made it so that the first request for an asset from any origin would be cached as a response to asset requests from all other origins. This caused subsequent requests from all other origins in the EU to fail due to Cross-origin resource sharing \(CORS\), resulting in the blank screen observed. **Mitigation:** The immediate impact was mitigated by invalidating the incorrect cached response in the EU instance of CloudFront, which restored Web and Mobile Experience access by 09:35 UTC. Additional assets-reindexing was performed to force any end-user mobile devices that had locally cached the incorrect responses to reload the Mobile Experience. **Recurrence Prevention:** The EU CloudFront cache policy configurations have been updated to match the US settings to prevent this incident from recurring.