Firstup Outage History

Started Feb 23, 2025, 08:55 AM UTC· resolved Feb 23, 2025, 01:23 PM UTC

1. investigating Feb 23, 2025, 08:55 AM UTC

   We are currently investigating an issue affecting campaign delivery to email and push notification channels for our US customers. Publishing to the feed and Notification Center is not affected. Content will continue to publish and display on web, mobile, and other endpoints as expected, and notifications are populating in the Notification Center. Customers in our EU communities (EU DC) are unaffected. We are actively working to identify the root cause and will provide an update within 30 minutes.

2. investigating Feb 23, 2025, 09:28 AM UTC

   We are continuing to investigate the issue affecting campaign delivery to email and push notification channels for our US customers. Our teams are actively working to resolve the issue as soon as possible. We will provide our next update in 30 minutes or sooner if there are significant developments.

3. investigating Feb 23, 2025, 10:30 AM UTC

   We are continuing to investigate the issue affecting campaign delivery to email and push notification channels for our US customers. We will provide further updates as soon as we have more information.

4. identified Feb 23, 2025, 11:38 AM UTC

   We are implementing a fix to attempt to mitigate the incident affecting campaign delivery. We'll provide further updates once the fix has been applied and we have monitored the service.

5. resolved Feb 23, 2025, 01:23 PM UTC

   The applied fix has restored operations, and campaign delivery has resumed. Any campaigns scheduled during the incident have now been sent and delivered successfully. This issue is now resolved. Thank you for your patience while we investigated and implemented a fix. If you experience any further issues, please contact Support at support.firstup.io

6. postmortem Mar 03, 2025, 03:55 PM UTC

   ## Summary: On Saturday, February 22nd, 2025, starting at 6:45 PM PT, our US-East RDS Database Cluster responsible for campaign delivery for customers hosted in the US entered into a crash loop during a database table repacking process, which forced the database to restart after every crash. This prevented platform communications from continuously being processed and sent out while the database restarted. ## Severity: Sev2 ## Scope: The scope of this service disruption was restricted to customers on the US platform utilizing the US-East database cluster. ## Impact: Some campaign communications may have experienced slight delivery delays via any delivery channel in the duration of this incident \(8hrs 27mins\). Customers on Employee Intranet solutions, Web Experience, Studio, custom API solutions, and Embedded content endpoints, would not have seen any impact. ## Root Cause: The root cause of this incident was attributed to a break in the integrity between a table in the database and its index, following a repacking task on the table to improve database performance. During the repacking of the table, its index was identified as having invalid indexes, and was skipped by the repacking process. The break in the integrity of the table and its index caused the database to enter into a crash loop starting at 6:45 PM PT, and restarted after every crash in an attempt to resolve the crash. In the periods between a successful database restart and the next database crash, other database processes were running as expected and campaign communications were sent out successfully. ## Mitigation: To mitigate this incident, the invalid index for the table was removed and recreated on Sunday, February 23rd, 2025, at 3:12 AM PT, in effect ending the crash loop on the database. ## Recurrence Prevention: To prevent this incident from recurring, we will have performed the following actions: * Checked all database indexes, and corrected any identified invalid indexes, including the offending index in this incident. * Added an additional parameter to the tables repacking task, which would skip repacking any tables if their indexes were found to be invalid in the future, thereby preserving their relational integrity, and avoiding a database crash.

Read the full incident report →

Started Feb 13, 2025, 06:15 PM UTC· resolved Feb 21, 2025, 08:51 PM UTC

1. investigating Feb 13, 2025, 06:15 PM UTC

   We are investigating reports where campaign communication deliveries on the EU and US platforms are delayed. This may include campaign emails, push notifications and assistant notifications. This is also causing and issue with the User Import feature as some uploads are stuck in processing state for an extended period of time.

2. investigating Feb 13, 2025, 06:15 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 13, 2025, 07:02 PM UTC

   The email delivery delays issue experienced today has been mitigated, and all email communications should have been delivered by now. We continue to investigate why some User Import jobs are stuck, and will provide another update within 1 hour.

4. monitoring Feb 13, 2025, 07:27 PM UTC

   The User Imports issue has also been mitigated, and all user import files are now processing as expected. We will be placing the impacted services under monitoring for now.

5. resolved Feb 21, 2025, 08:51 PM UTC

   All impacted systems have remained available and fully functional during the monitoring phase of this service degradation. This incident is now resolved.

6. postmortem Mar 05, 2025, 05:22 PM UTC

   ## Summary: On Thursday, February 13th, 2025, reports of delays in campaign email deliveries, as well as processing failures associated with user import files were received via tickets to our Customer Support Center. Following a correlation of the number of tickets received and the nature of the issues reported, a platform incident was declared at 10:02 AM PT and an incident response team began investigating the reports. A service degradation notification was subsequently published on our status page at 10:15 AM PT. Investigations revealed that the reported issues were not related in any way, and troubleshooting of each issue continued independently and concurrently during the incident. ## Severity: Sev2 ## Scope: Campaign communication delays: * Customers on the Firstup platform with email campaigns published on 2/13/2025 between 7:00 AM PT and 7:40 AM PT. User import/sync issue: * Customers on the Firstup platform who uploaded user sync and user import files on 2/13/2025 between 2:09 AM PT and 11:22 AM PT. ## Impact: * Campaigns in the scope above may have experienced email delivery delays of 20 to 30 minutes. * User import and user sync files in the scope above may have returned processing errors or were stuck in a processing state for up to 9hrs 13mins. All other platform services remained uninterrupted during this incident. ## Root Causes: Campaign communication delays: * A long-running query in the database created a bottleneck effect, resulting in a slowdown in processing queries associated with the campaigns in scope of the incident until the job completed at 7:40 AM PT. User import/sync issue: * Following a configuration change in the backend intended at improving Multi-Program user membership at 2:09 AM PT, user re-indexing activity started to surge as customers uploaded their user sync files, which overloaded the indexing queue and caused delays in bulk uploads and snapshot creation. ## Mitigation: Campaign communication delays: * No mitigation steps were taken to resolve the campaign communication delays, as the long-running query completed on its own, eliminating the slowdown in processing normal campaign publishing queries. User import/sync issue: * The offending configuration change was internally identified and reverted at 9:36 AM PT. However, the messages in queue to be processed had surged so much that additional processing resources were added at 10:43 AM PT to aid in draining the queue. The queue returned to normal levels by 11:22 AM PT. ## Recurrence Prevention: To prevent a recurrence of the issues in this incident due to the same root cause, we have performed or have committed to perform the following actions: Campaign communication delays: * Created a monitor that will alert on long-running queries, to allow for faster intervention before a platform incident happens. * Move the offending query to a separate data pipeline not utilized by the campaign delivery process. * Add Statement and Deadlock timeouts to the campaign delivery processing database to reduce the impact of long-running queries. User import/sync issue: * Add monitoring and alerting for User-Sync and Bulk-Upload errors on a platform level. * Evaluate permanently increasing processing resources to account for unforeseen surges in the messages queue. * Review front-end UI error reporting messaging to be more intuitive.

Read the full incident report →

Started Jan 22, 2025, 11:25 PM UTC· resolved Jan 30, 2025, 08:53 PM UTC

1. investigating Jan 22, 2025, 11:25 PM UTC

   We are investigating reports where campaign communication deliveries on the US platform are slightly delayed. This may include campaign emails, push notifications, or assistant notifications. An update will be provided with 1 hour.

2. investigating Jan 23, 2025, 12:26 AM UTC

   As we continue to investigate this incident, we can confirm that campaign communications continue to be delivered, albeit in a slightly delayed fashion. Another update within 1 hour.

3. monitoring Jan 23, 2025, 12:50 AM UTC

   We identified that a long-running query overutilized the database's processing power, therefore slowing down the delivery of other campaign communications. This has now been mitigated, and the database is back to a normal state, with all communication being processed as expected. We will be placing the affected services under monitoring for now.

4. resolved Jan 30, 2025, 08:53 PM UTC

   This service degradation is now resolved, and all impacted services are fully available and functional.

5. postmortem Jan 31, 2025, 09:56 PM UTC

   **Summary:** On Wednesday, January 22nd, 2025, starting at around 2:22 PM PT, we started receiving reports of delays in the delivery of campaign emails to end users. After a correlation of these reports and system monitors and alerts, a platform incident was declared at 3:14 PM PT, and published in our status page. **Severity:** Sev2 **Scope:** The scope of this service degradation included any communications \(campaign emails, push notifications, assistant messages\) on the US platform. **Impact:** During the service degradation, messages delivery on the US platform was delayed for a few minutes - up to 2.5 hours. These included campaign emails, push notifications, and assistant messages. **Root Cause:** The incident response team identified the root cause of the incident to be a long-running query that caused an over-utilization of the responding database’s resources. As a result, other database requests remained in queue, and were processed as database resources became available. **Mitigation:** To mitigate this incident, the long-running query was disabled at 3:30 PM PT, in effect freeing up database resources. This allowed for the messages that were in queue to be processed and delivered. System monitors confirmed normal system performance at 4:39 PM PT, and the incident was declared as resolved at 4:43 PM PT after the database queue was confirmed as caught up. **Recurrence Prevention:** The following actions have been taken or identified as follow-up items to prevent a recurrence of this incident: * The offending query has been optimized to utilize minimal database resources. * Additional database timeout parameters will be added to better address long-running queries that result in resource contention. * Enhanced the existing system monitors and alerts to include links to runbooks to aid in quickly mitigating similar issues in the future.

Read the full incident report →

Started Jan 21, 2025, 08:58 PM UTC· resolved Jan 24, 2025, 03:27 PM UTC

1. investigating Jan 21, 2025, 08:58 PM UTC

   We are currently investigating reports where the Employee Experience is currently inaccessible due to some sort of maintenance.

2. monitoring Jan 21, 2025, 09:04 PM UTC

   This incident is now resolved. Placing the Employee Experience under monitoring for now.

3. resolved Jan 24, 2025, 03:27 PM UTC

   The impacted components have remained stable and fully available. This incident is now resolved.

4. postmortem Jan 24, 2025, 03:27 PM UTC

   **Summary:** On Tuesday, January 21st, 2025, starting at around 12:35 PM PT, we started receiving reports that the Employee Experience was not visible, but rather, a “Under Construction” notice displayed in its place. Users were unable to close the notice and proceed to the Employee Experience. Initial investigations indicated that the reported issue was further reaching than just the customers who had reported it, and a platform incident was declared and published on the Firstup status page at 12:58 PM PT. **Severity:** Sev2 ‌ **Scope:** The scope of this service disruption included all end users on the Firstup platform who attempted to access the Employee Experience in the duration of this incident. ‌ **Impact:** In the duration of this incident \(30mins\), end users could successfully log into the Employee Experience but saw a “Under Construction” notice instead of their expected Employee Experience frontpage and could not navigate away from the notice overlay. The Employee Experience was still available behind the Pendo guide overlay, and no data was compromised as a result of the incident. ‌ **Root Cause:** The incident response team quickly identified the root cause of the incident to be a Pendo guide overlay that was erroneously published by a Firstup employee across the entire Employee Experience platform at 12:29 PM PT. This guide was intended for a specific customer account, which was not correctly tagged during its publication, and therefore, visible to our entire customer base. Pendo is a third party tool that allows for in-app annotations and user messaging that is primarily used by the Product Management and Marketing organizations primarily targeting program managers actively using Creator Studio. **Mitigation:** To resolve this incident, the Pendo guide was deactivated at 12:59 PM PT, which restored visibility to the Employee Experience immediately. **Recurrence Prevention:** The following actions have been identified as follow-up items to prevent a recurrence of this incident: * Implement a Pendo guide review process before publication. Guides will no longer be immediately able to launch without a review from an authorized publisher. * An audit of all existing guides currently Published should also be carried out, to check for necessary usage and alignment to current standards. * An audit of associate permissions in Pendo to identify who should and should not be able to publish Pendo guides.

Read the full incident report →

Started Jan 01, 2025, 03:50 PM UTC· resolved Jan 01, 2025, 06:57 PM UTC

1. investigating Jan 01, 2025, 03:50 PM UTC

   We are investigating a report of a service disruption affecting campaign email deliverability. Emails are being delivered however they may be quarantined for some customers as a result of being sent from shared SendGrid IP addresses instead of our dedicated domain and IP addresses. We are working with SendGrid to resolve.

2. monitoring Jan 01, 2025, 05:31 PM UTC

   This issue has now been mitigated Action has been taken to send all emails from our dedicated domain and IP addresses and not from shared SendGrid IP addresses. We are currently monitoring to ensure emails continue to be sent successfully.

3. resolved Jan 01, 2025, 06:57 PM UTC

   This incident has been resolved.

4. postmortem Jan 13, 2025, 04:16 PM UTC

   **Summary:** On Wednesday, January 1st, 2025, starting at around 6:41 AM PT, we received reports that some campaign emails were not being delivered as expected. While some campaign emails were being delivered successfully, others were reported as being quarantined for customers with strict email security policies. A platform incident was declared and published at 7:50 AM PT, and an incident response team began investigating these reports immediately. **Severity:** Sev2 **Scope:** The scope of this service degradation was restricted to Firstup customers with strict email security policies that did not allow email deliveries originating from IP addresses other than the already allow-listed Firstup dedicated sender IP addresses. Other campaign delivery channels, such as Push Notifications and the Assistant, were not affected by this incident. **Impact:** In the duration of this incident \(16hrs 46mins\), customers with strict email security policies may have observed that some campaign emails were being silently dropped, quarantined, or bounced because they originated from IP addresses other than the already allow-listed Firstup dedicated sender IP addresses. **Root Cause:** The root cause was determined to be a misconfiguration on Firstup’s account by our third-party email delivery vendor, where, on December 31st, 2024 at 4:38 PM PT, a Technical Account Manager \(TAM\) on the vendor’s side erroneously placed Firstup’s dedicated email sender IP addresses in a “warming” state. This reduced the number of emails each dedicated IP address could send daily. Once that threshold was reached on January 1st, 2025, the remaining campaign emails were sent out from other IP addresses that were not associated with Firstup, which triggered a security response from customers with strict email security policies \(as designed and expected\), resulting in those emails either being silently dropped, quarantined, or bounced. **Mitigation:** To resolve this incident, Firstup’s dedicated sender IP addresses were removed from the “warming” state at 9:24 AM PT, which restored their full capabilities and normal throughput. **Recurrence Prevention:** The following actions have been taken or committed to by our third-party email delivery vendor: * Have set up coaching sessions with the TAM team, to ensure better understanding of this type of procedure. * Confirmed that the TAM in question will not be on Firstup’s account again. * General reinforcement on account change procedures will be provided to all our TAMs to ensure this is followed up properly.

Read the full incident report →

Started Dec 19, 2024, 08:36 PM UTC· resolved Dec 19, 2024, 11:07 PM UTC

1. investigating Dec 19, 2024, 08:36 PM UTC

   We are investigating a reports of a service disruption affecting campaign email deliverability. At this time, some campaign emails, test emails and user file success emails may not be sending/delivering successfully. Our next update will be in 30 minutes.

2. investigating Dec 19, 2024, 09:40 PM UTC

   Update: Incident scope currently appears to be limited to a single email security provider, Proofpoint, who is currently blocking some of the Firstup sender IPs. We have reached out to that vendor directly and will be following up with Customers asking that they request the Firstup sender IPs be unblocked since they are affecting email delivery to their employees and end users.

3. monitoring Dec 19, 2024, 09:52 PM UTC

   Sending IP addresses have been unblocked by Proofpoint. Work is ongoing now to resubscribe affected email addresses from delivery failures. Reducing impact to minor and component performance to degraded.

4. resolved Dec 19, 2024, 11:07 PM UTC

   No further errors have been observed for email sent to Proofpoint MX endpoints and we do not see evidence of widespread unsubscribed users. Marking incident as resolved, and all components fully operational.

5. postmortem Jan 15, 2025, 06:00 PM UTC

   **Summary:** Shortly after 11:30AM PT on December 19th, 2024 Firstup began to receive Customer reports of emails not being received. An incident was declared an hour later after it was determined Proofpoint’s dynamic reputation engine had flagged all three Firstup production sending IPs, resulting in many emails being blocked by the recipient email server\(s\). This observation was shared back with Customers impacted so that they could request that IPs be removed from Proofpoint’s list. Firstup also requested the same through Proofpoint’s public facing page. The IPs were confirmed to have been removed from the blocklist by 1:52PM PT which immediately mitigated impact from further emails being negatively affected. **Impact:** All email in scope of the incident was not delivered to the intended recipients, nor was there an attempt to redeliver the original emails due to the nature of the error Proofpoint email servers sent back to Firstup. Proofpoint was responding to all emails from these IPs with the following error: * 554 Blocked No users were unsubscribed as a result of the delivery error, so any email sent to the same users after the sending IPs were removed from Proofpoint’s blocklist were successfully received. However, any emails that were blocked and not sent using optimized delivery with retargeting, had to be resent through Creator Studio as they were not quarantined or deferred by the email systems. Longer running campaigns configured to include retargeting, would have continued to function beyond the day of the incident and likely resent emails that were originally blocked. **Root Cause:** Root cause was determined to be related to Proofpoint’s Dynamic IP Reputation Engine. There are a small number of triggers that Proofpoint discloses publicly in their FAQ \([https://www.proofpoint.com/us/support-services/ip-blocked-faq](https://www.proofpoint.com/us/support-services/ip-blocked-faq)\) that Firstup has confirmed did not occur. Because the inner workings of this system are proprietary, and Proofpoint has not provided any specific remediation steps for tickets opened to remove the IPs from the list, our conclusion is that a false positive indicator on this system misidentified the sending IPs. Firstup sends hundreds of thousands of employer to employee emails per hour to Proofpoint servers on any given day, and December 19th was no different. **Mitigation:** Impact was mitigated through a multi-pronged approach. Firstup used Proofpoint’s public-facing removal request system at [https://ipcheck.proofpoint.com/](https://ipcheck.proofpoint.com/) and also worked through Customer IT and email security contacts to request the removal using Proofpoint’s expedited removal request system. One or both of these measures resulted in the three affected sending IP addresses being removed within the hour after the requests were made. **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * Customers affected by the incident have been encouraged to open their own ticket with Proofpoint, as well as confirm that the three sending IP addresses have been allowlisted on their own Proofpoint instance and have a higher priority than the PDR blocklist. Further information on allowlisting email can be found in the following Knowledge Article - [https://support.firstup.io/hc/en-us/articles/4417455533975-Allowlist-Emails-from-Firstup](https://support.firstup.io/hc/en-us/articles/4417455533975-Allowlist-Emails-from-Firstup) * More granular monitoring is being put into place to identify patterns of throttling, errors, and delivery failures by program ID, and an outreach program will be established to work with Customers that appear to be misconfigured, or whose email systems cannot handle the volume of email they are attempting to send at any given time.

Read the full incident report →

Started Nov 24, 2024, 12:00 AM UTC· resolved Nov 24, 2024, 12:00 AM UTC

1. resolved Nov 24, 2024, 12:30 AM UTC

   Service restoration did not occur within the expected window.

Read the full incident report →

Started Nov 13, 2024, 03:07 PM UTC· resolved Nov 16, 2024, 02:03 AM UTC

1. investigating Nov 13, 2024, 03:07 PM UTC

   We are currently investigating reports of intermittent Studio slow performance issues. We will provide you with an update within 1 hour.

2. identified Nov 13, 2024, 03:30 PM UTC

   We have identified a potential root cause of this incident, and are working towards mitigation. Affected components are only on the US datacenter. The EU datacenter remains unaffected. Another update within 1 hour.

3. identified Nov 13, 2024, 04:25 PM UTC

   A hotfix for this incident has been developed and is currently being tested in our staging environment. Once vetted, it will be released in the production environment. Another update within 1 hour.

4. monitoring Nov 13, 2024, 04:35 PM UTC

   The hotfix for this incident has been released in the production environment. All services should now be restored. We will be placing the affected services in a monitoring state for now.

5. identified Nov 13, 2024, 05:50 PM UTC

   We are currently investigating a residual impact of this incident where some email campaigns are yet to be delivered after the hotfix was released. We will provide an update within 1 hour.

6. identified Nov 13, 2024, 06:47 PM UTC

   We have identified a potential root cause for the residual impact of this incident where email campaigns are yet to be delivered following the hotfix. We are working on developing and verifying another hotfix for that issue. Another update within 1 hour.

7. identified Nov 13, 2024, 07:48 PM UTC

   We continue working on developing and verifying another hotfix for the email delivery issue. Another update within 1 hour.

8. monitoring Nov 13, 2024, 08:01 PM UTC

   The hotfix for the campaign email delivery issue has now been vetted and deployed in the production environment. Campaign emails have now resumed delivery and may take a few minutes to hit the end user’s inbox. The affected services will now be placed back under monitoring for now.

9. resolved Nov 16, 2024, 02:03 AM UTC

   All impacted Firstup platform endpoints have remained stable and fully available. This incident is now resolved.

10. postmortem Nov 16, 2024, 02:03 AM UTC

    ## Summary: On Wednesday, November 13th, 2024, starting at around 6:14 AM PT, we received reports of some users experiencing a general slowness and decreased responsiveness while navigating around Firstup’s Studio platform. Additional reports of some published campaign emails not being delivered, and error messages being returned while navigating within the Employee Experience followed shortly thereafter. ## Severity Sev2 ## Scope: The scope of this service degradation included all Firstup customers utilizing the Studio and the Employee Experience \(EE\) endpoints. ## Impact: For approximately 2.5 hours after the onset of the incident, users were unable to efficiently and consistently navigate around Studio and the EE. Symptoms included a loading spinner for an extended amount of time in Studio, and missing shortcuts in EE, with some of the errors noted below being returned: * Oops, an error occurred! Sorry about that. * TypeError: toMoment\(…\).format is not a function Concurrently, campaign emails were also not being delivered consistently to their intended recipients, and user sync files were not being actively processed. Most campaign emails were delivered over the course of the next 3 hours with a few rare exceptions that required manual intervention and Customer coordination. User sync was not restored until it was discovered the process was not actively running the following day. Total incident duration less the user sync process was just over 6 hours. ## Root Cause: The root cause has been attributed to a code change that was released during the Platform Software Release maintenance window the night before, which caused a slow-running query from one of our back-end services to run for too long and utilize an exceptionally large amount of database resources including the number of network connections and CPU processing cycles. This, coupled with the normal increase in database requests from our customer base during our platform utilization peak hours starting at around 5:54 AM PT, resulted in available database connections to be exhausted and CPU overutilization conditions in the database. As a result, new connections to the database could not be freely established until current connections were closed and made available for new requests from various platform services such as Studio and EE. The backend service responsible for campaign email delivery was also subject to this condition and could not process email deliveries as expected. Similarly, user sync file processing was also delayed beyond normal limits. ## Mitigation: Various symptoms exhibited during the course of the incident were mitigated in phases. The most significant service impact was mitigated after a hotfix was released at 8:29 AM PT to halt the aforementioned slow-running query, in effect relieving some resource pressure on the database, and allowing customer-facing service requests from Studio and EE to successfully re-establish connections with the database. Additional resources were also spun up to process the email delivery queue backlog that had been increasing during the incident, which started draining at 12:02 PM PT. Almost all campaign emails were confirmed delivered by 1:10 PM PT, with the exception of ~150 email messages, which had experienced an internal error, but were later cleared and delivered. ## Recurrence Prevention: The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * Moved the slow-running query to run during after-hours \(off customer peak hours\). * Isolate the campaign email delivery back-end service to its own database cluster to avoid the general database impact on email deliveries. * Enhance our software change management policies to release risky backend changes behind a feature flag for a more controlled release. * Update post incident service verification to ensure that user sync processing has been fully restored and remains functional.

Read the full incident report →

Started Nov 04, 2024, 04:40 PM UTC· resolved Nov 12, 2024, 05:36 PM UTC

1. investigating Nov 04, 2024, 04:40 PM UTC

   We are investigating reports where some email campaigns are not being delivered as expected. We will provide you with an update within 1 hour.

2. investigating Nov 04, 2024, 05:11 PM UTC

   As we continue to investigate these reports, our current observation is that email campaigns are being delivered, albeit with some delays. Another update within 1 hour.

3. investigating Nov 04, 2024, 06:11 PM UTC

   Our investigations have not revealed any issues on our platform. However, we see a potential issue with our upstream third-party email delivery vendor, and have reached out to them for additional troubleshooting on their end. We will provide you with another update within 1 hour.

4. investigating Nov 04, 2024, 07:11 PM UTC

   We continue to work with our upstream third-party email delivery vendor towards a solution to this issue. Another update within 1 hour.

5. investigating Nov 04, 2024, 08:11 PM UTC

   We continue to work with our upstream third-party email delivery vendor towards a solution to this issue. Another update within 1 hour.

6. identified Nov 04, 2024, 09:23 PM UTC

   As we continue to work with our upstream third-party email delivery vendor towards a solution to this issue, we have identified that platform emails such as password reset emails are in the scope of this incident. Another update within 1 hour.

7. monitoring Nov 04, 2024, 11:24 PM UTC

   This issue has now been mitigated, and any emails that had not been delivered should now be hitting their user endpoints. We will place the impacted services under monitoring.

8. resolved Nov 12, 2024, 05:36 PM UTC

   Email delivery has remained available and fully functional throughout the monitoring phase of this incident. This incident is now resolved. Once available, a Root Cause Analysis for this incident will be published here.

9. postmortem Nov 13, 2024, 01:23 AM UTC

   **Summary:** On Monday, November 4th, 2024, starting at around 7:53 AM PT, we received reports that some published campaign emails were not being delivered to their intended audiences. While some emails were delivered as expected, others were either delayed or appeared to be stuck. All emails were handed off in a timely manner from the Firstup platform to the third-party email provider. This problem worsened over the course of the next several hours where email throughput appeared to be highly restricted, while the backlog of the email delivery queue not only continued to grow, but also did not drain in a logical chronological order in which messages were initially queued. Through a joint troubleshooting call with the third-party email provider, it was determined that a large volume of email delivery errors starting at around 7:11 AM PT had put the entire pool of our email delivery IP addresses in a state of reduced performance. After jointly reviewing the highest volume of errors with the third-party email provider, the sender IPs were restored to a fully functioning state, resulting in the entire email backlog being drained fully by 3:30 PM PT. **Severity:** Sev2 **Scope:** The scope of this service degradation was restricted to customers who use Firstup campaign email delivery as a channel, as well as any other non-campaign email content sent from the Firstup platform, such as password reset request emails. Push notifications, assistant notifications, and web or mobile experience channels were unaffected and remained fully functional. **Impact:** Within an individual campaign sent to email as a channel, some emails may not have been delivered as expected, while others wound up being stuck in a "processing" state on the third- party email delivery platform. During the incident \(7hrs 58mins\), some of the emails in the “processing” state were successfully delivered but heavily delayed, while others remained stuck in a “processing” status. Observed email throughput was reduced to approximately 46k messages per hour from a theoretical max of 30k messages per second. The total outstanding backlog prior to mitigation was over a million email messages. **Root Cause:** Root cause has been attributed to an elevated level of email delivery errors that triggered a protection mechanism on the third-party email provider platform. This resulted in reduced throughput for the entire pool of our sender IP addresses to the point where mostly retries for deferrals from earlier delivery errors were being processed, and very few queued up emails were delivered. Essentially the queue processing equivalent of running in place. **Mitigation:** After analyzing the top contributors to email delivery errors that appeared to be correlated to a single misconfigured email security endpoint all addresses associated with that endpoint were force-unsubscribed until it could be correctly configured, to avoid any further email delivery errors contributing to the underlying log jam. 80k email errors were attributed to that endpoint in just a couple of hours. Through a joint incident bridge with the third-party email provider, Firstup demonstrated the irrelevance of the deferral rates to the overall email backlog queue. A data pipeline engineer was paged out and able to verify that the sender IPs had been relegated to a lower performing state that was actually contributing to a circular problem. Backend system changes were made at 3:09 PM PT on the third-party platform to restore prior state of the sender IPs, and the entire email messages backlog subsequently fully drained in less than 25 minutes. **Recurrence Prevention:** The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: * Email addresses contributing to elevated error rates will be bulk unsubscribed from the platform \(or otherwise quarantined\) until underlying conditions can be corrected. * Coordinate with third-party provider to better understand the characteristics of the platform safety mechanism, including why it triggered, how to avoid it entirely, and how to improve joint monitoring and mitigate elevated error rates from affecting overall delivery. * Implement any reasonable recommendations from the third-party RFO.

Read the full incident report →