Firstup incident
Platform Service Degradation - Email Delivery Inconsistencies
Resolved after 8 hours 29 minutes, affecting Campaign Delivery, on June 22, 2026.
## Summary: On June 22nd, 2026, we started receiving reports where email campaigns were being delivered to only a subset of the targeted email audience. Following initial triaging and troubleshooting, a platform incident was declared, and the incident response team began looking at the reported issue. The unexpected behavior was determined to be due to a logic error in user reachability through the email channel, and the software change implicated was immediately rolled back to resolve the issue. ## Severity: Sev-2 ## Scope: Some campaigns sent between 12:54 AM PT on June 20th, 2026, and 12:59 PM PT on June 22nd, 2026, to an email audience from the Firstup platform. ## Impact: Emails were not sent to the entire email audience for campaigns sent during the incident duration. ## Root Cause: The root cause for this incident was determined to be a regression from a software change that was made on June 20th, 2026, to align with the reachability definitions that we introduced in the Q2 quarterly release. A software bug was identified that returned a “Unknown” email address reachability status when an email audience was being processed for campaign delivery in the backend, resulting in those email addresses being excluded from the campaign’s delivery backend path. ## Mitigation: To immediately mitigate this incident, a rollback of the software change was performed, which was completed at 12:59 PM PT, restoring the reachability data to its previous state platform-wide. For customers who had campaigns in scope of this incident, and needed to send the email to users who were initially excluded from the campaign’s delivery, our Support team provided a list of those users to be retargeted separately via a duplicate campaign. ## Recurrence Prevention: Our recurrence prevention measures include: * Investigate and fix the software bug that returned the “Unknown” email address reachability status. * Update our QA processes to include robust end-to-end testing on all New Product Features and Enhancements before being released.
Affected components
Update timeline
- investigating Jun 22, 2026, 05:11 PM UTC
We are investigating reports where campaign emails are not being delivered to the entire targeted campaign audience. We will provide you with an update within 1 hour.
- identified Jun 22, 2026, 05:48 PM UTC
We have identified a code change that resulted in the observed behavior. We are currently rolling back that code change and will provide another update once that is completed.
- identified Jun 22, 2026, 06:51 PM UTC
We are still in the process of rolling back the code change we believe caused the observed behavior. We anticipate this should be completed soon, but we will provide another update on its progress within the next hour.
- identified Jun 22, 2026, 07:11 PM UTC
We currently estimate that the code change rollback will be completed within the next 2 hours, which will fully mitigate the impact of the observed behavior. We will provide another update as soon as this is completed.
- identified Jun 22, 2026, 08:09 PM UTC
We have completed the code change rollback, and this incident is now mitigated. We will be placing the impacted component(s) under active monitoring to ensure continued proper functionality.
- monitoring Jun 22, 2026, 08:09 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 23, 2026, 01:41 AM UTC
This incident is now resolved and all components marked fully operational. No further issues observed while monitoring since the platform incident was mitigated at 1PM PT today.
- postmortem Jul 01, 2026, 04:37 PM UTC
## Summary: On June 22nd, 2026, we started receiving reports where email campaigns were being delivered to only a subset of the targeted email audience. Following initial triaging and troubleshooting, a platform incident was declared, and the incident response team began looking at the reported issue. The unexpected behavior was determined to be due to a logic error in user reachability through the email channel, and the software change implicated was immediately rolled back to resolve the issue. ## Severity: Sev-2 ## Scope: Some campaigns sent between 12:54 AM PT on June 20th, 2026, and 12:59 PM PT on June 22nd, 2026, to an email audience from the Firstup platform. ## Impact: Emails were not sent to the entire email audience for campaigns sent during the incident duration. ## Root Cause: The root cause for this incident was determined to be a regression from a software change that was made on June 20th, 2026, to align with the reachability definitions that we introduced in the Q2 quarterly release. A software bug was identified that returned a “Unknown” email address reachability status when an email audience was being processed for campaign delivery in the backend, resulting in those email addresses being excluded from the campaign’s delivery backend path. ## Mitigation: To immediately mitigate this incident, a rollback of the software change was performed, which was completed at 12:59 PM PT, restoring the reachability data to its previous state platform-wide. For customers who had campaigns in scope of this incident, and needed to send the email to users who were initially excluded from the campaign’s delivery, our Support team provided a list of those users to be retargeted separately via a duplicate campaign. ## Recurrence Prevention: Our recurrence prevention measures include: * Investigate and fix the software bug that returned the “Unknown” email address reachability status. * Update our QA processes to include robust end-to-end testing on all New Product Features and Enhancements before being released.
Your product runs on more than Firstup
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services