Drip incident
Disruption of Service - Links and Page Loading Issues
Drip experienced a major incident on November 25, 2020 affecting REST and JavaScript APIs and User Interface and 1 more component, lasting 21h 13m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 25, 2020, 03:11 PM UTC
A key service provider is currently experiencing issues which is directly impacting Drip. We are getting reports of pages not loading and links within emails timing out. Our team is investigating these issues and we will post an update once we have more information available.
- investigating Nov 25, 2020, 03:49 PM UTC
We are continuing to work on ways to mitigate this issue impacting links and pages loading properly. While we will continue to provide updates here as we make progress, do not hesitate to contact us at [email protected] if you have any questions. We are working to get this issue resolved as quickly as we can.
- identified Nov 25, 2020, 04:41 PM UTC
We've deployed an updated that has improved links and clicks loading successfully, though we are still seeing some failures come through. Our team will continue to work on additional fixes during this service disruption, and any updates will be posted immediately. Please continue to reach out to our Support Team with any additional questions via [email protected].
- identified Nov 25, 2020, 05:18 PM UTC
Our team has identified some additional fixes that we are working on putting in place shortly to help alleviate some of the issues we are seeing with clicks and pages loading. Our Support Team over at [email protected] is ready to answer any questions that you may have while we continue to make these changes. Thanks for sticking with us through this.
- identified Nov 25, 2020, 05:43 PM UTC
We are happy to report that the recently deployed fix has resolved the issue with links in emails leading to error pages. We are seeing event delays due to the ongoing service disruption, and will continue to post updates as we work through them. Please get in contact with our Support Team over at [email protected] if you are still seeing issues, and thank you again for your patience while we make these adjustments.
- investigating Nov 25, 2020, 07:15 PM UTC
We are currently seeing delays with events, automation emails, confirmation emails, and segmentation updates due to the ongoing service disruption with Amazon Web Services. This may also cause API calls to intermittently fail. Broadcast emails and campaign emails are sending normally, but segmentation (such as resend to un-open criteria) may not be fully up to date. This may mean that more people than intended may get a broadcast resend email in some cases. Events will be queued once this disruption is resolved, and we are working on restoring service as quickly as possible via alternative hosting. Please continue to reach out to us at [email protected] if you see any other issues, and we will post updates as soon as we have them.
- identified Nov 25, 2020, 08:18 PM UTC
We are still seeing delays with events, confirmation emails, and segmentation updates due to the ongoing service disruption with Amazon Web Services. This also is causing API calls to intermittently fail. Our processing of automations, including automation emails, are similarly delayed. As of right now, we expect the events included in automations, triggers, delays, emails, goals, etc to run in chronological order once this disruption is over Our support team is reachable via [email protected] for any questions, concerns, or help if needed. Thank you for your ongoing patience while we work through these issues.
- identified Nov 25, 2020, 09:20 PM UTC
Our team is still working to resolve these issues from the ongoing service outage. We will provide additional updates hourly unless we have news before that time.
- identified Nov 25, 2020, 10:20 PM UTC
There are no changes to report for the moment, but we're continuing our work toward a resolution. We will update you again here soon.
- identified Nov 25, 2020, 11:25 PM UTC
We do not have any new updates at this time, and our team is still working to mitigate this outage. Please reach out to support at [email protected] with any questions while we work through this. Thank you for sticking with us.
- identified Nov 26, 2020, 12:26 AM UTC
We are starting to process our backlog of delivery analytics data (clicks and opens) and hope to have more data flowing through soon. We will keep this page up to date as we get more info about this ongoing outage.
- identified Nov 26, 2020, 01:32 AM UTC
We are continuing to work through backlog of clicks and open data, but have no further updates at this time. We'll be back with an more information as we have it for you.
- identified Nov 26, 2020, 02:38 AM UTC
We are processing data for our backlogged open and click data more quickly and should see that updating in app now. We are still seeing issues with other impacted portions of the app due to the outage, and hope to have more information on this soon as we are staring to see updates about this outage turning the corner. Thank you again for patience.
- monitoring Nov 26, 2020, 07:08 AM UTC
We are happy to report that we are seeing all services are back online. We are still seeing delays due to the large backlog of data from the outage, but will push that data through as quickly as we can. We will post another update once everything is fully caught up. Thank you so much for your patience while we got things running again.
- resolved Nov 26, 2020, 12:24 PM UTC
We are happy to say all functions are working as normal and data is fully up to date within the app. Thank you again one last time for your patience during these issues. If you are still seeing any issues from the past day, please reach out to our Support Team via [email protected] and we'll be happy to assist.