Firstup incident

Platform Service Unavailable - Insights Reports

Resolved after 3 days 22 hours, affecting Insights, on October 9, 2025.

postmortem Oct 22, 2025, 01:48 PM UTC

## Summary: On October 9th, 2025, starting at around 10:03 AM PT, we received reports where Studio users received a “You don’t have access to Insights Reports” error notification while trying to access Studio Insights Reports. A platform incident was declared at 10:23 AM PT and published on our Status Page at 10:29 AM PT, as the incident response team investigated the issue. ## Severity: Sev-2 ## Scope: All Studio users for customers on the US Firstup platform attempting to access Studio Insights Reports were in scope of this incident. ## Impact: Any Studio user on the US Firstup platform observed a “You don’t have access to Insights Reports” error notification while trying to access Studio Insights Reports in the duration of this incident \(59 minutes\), rendering the reports unavailable to the user. Customers on the EU Firstup platform were not impacted by this incident. ## Root Cause: The root cause for this incident was determined to be user error when a Firstup engineer accidentally added the program IDs for customers on the US Firstup platform to the Insights Reports blocked list \(which takes a priority over the allowed list\) in the production database. This was a manual process and a cut and paste error. ## Mitigation: To mitigate this incident, the latest production database snapshot was restored, and the Insights Reports microservice was repointed to the now-good snapshot at 11:02 AM PT, restoring Insights Reports access to Studio users. ## Recurrence Prevention: To prevent this incident from happening again, we have implemented a peer review process for any blocked programs additions and subtractions as part of our Insights Reports production configuration change management.

Started
Oct 09, 2025, 05:29 PM UTC
Resolved
Oct 13, 2025, 03:58 PM UTC
Duration
3d 22h
Detected by Pingoru
Oct 09, 2025, 05:29 PM UTC

Affected components

Insights

Update timeline

  1. investigating Oct 09, 2025, 05:29 PM UTC

    We are currently investigating an issue where Studio Insights Reports are not available on the Firstup US platform. We will provide you with an update within 1 hour.

  2. monitoring Oct 09, 2025, 06:05 PM UTC

    We have mitigated this incident, and Insights Reports on the Firstup US platform are now available. A screen refresh may be required to load the reports. We will be placing the impacted components under monitoring for now.

  3. resolved Oct 13, 2025, 03:58 PM UTC

    This incident is now resolved, and all impacted services have remained available and fully operational. A Root Cause Assessment will be provided here once a postmortem review of this incident is concluded.

  4. postmortem Oct 22, 2025, 01:48 PM UTC

    ## Summary: On October 9th, 2025, starting at around 10:03 AM PT, we received reports where Studio users received a “You don’t have access to Insights Reports” error notification while trying to access Studio Insights Reports. A platform incident was declared at 10:23 AM PT and published on our Status Page at 10:29 AM PT, as the incident response team investigated the issue. ## Severity: Sev-2 ## Scope: All Studio users for customers on the US Firstup platform attempting to access Studio Insights Reports were in scope of this incident. ## Impact: Any Studio user on the US Firstup platform observed a “You don’t have access to Insights Reports” error notification while trying to access Studio Insights Reports in the duration of this incident \(59 minutes\), rendering the reports unavailable to the user. Customers on the EU Firstup platform were not impacted by this incident. ## Root Cause: The root cause for this incident was determined to be user error when a Firstup engineer accidentally added the program IDs for customers on the US Firstup platform to the Insights Reports blocked list \(which takes a priority over the allowed list\) in the production database. This was a manual process and a cut and paste error. ## Mitigation: To mitigate this incident, the latest production database snapshot was restored, and the Insights Reports microservice was repointed to the now-good snapshot at 11:02 AM PT, restoring Insights Reports access to Studio users. ## Recurrence Prevention: To prevent this incident from happening again, we have implemented a peer review process for any blocked programs additions and subtractions as part of our Insights Reports production configuration change management.

Your product runs on more than Firstup

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →