Total Expert experienced a minor incident on February 13, 2023 affecting API and Email Messages and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 13, 2023, 07:08 PM UTC
Some users may be experiencing an issue where system activities are not populating in the user's activity stream.
- investigating Feb 13, 2023, 07:29 PM UTC
We are preparing to perform emergency database maintenance in order to resolve the reported issue.
- identified Feb 13, 2023, 08:14 PM UTC
We are currently performing emergency database maintenance.
- identified Feb 13, 2023, 08:58 PM UTC
We are continuing to perform emergency database maintenance. We will provide another update in 1 hour.
- identified Feb 13, 2023, 09:56 PM UTC
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Services impacted include journeys, imports, and email sending. Some pages may display stale data during this window. We will provide another update in 1 hour.
- identified Feb 13, 2023, 11:04 PM UTC
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window. We expect to have paused services operational by 7pm Central. We will provide another update in 1 hour.
- identified Feb 13, 2023, 11:59 PM UTC
We are continuing to perform emergency database maintenance. During this maintenance, all background processing is disabled. Paused services include journeys, imports, and email sending. Some pages may display stale data during this window. We are still on schedule to have paused services operational by 7pm Central.
- monitoring Feb 14, 2023, 01:42 AM UTC
All Functionally has been restored. Emergency database maintenance has now been completed for everything but Activity Logs and Tasks. Activity Logs and Tasks are being actively restored and will be appearing over the next few hours and days. Contact list updates may be delayed until we finish processing our backlog of events. We will continue to monitor the progress and provide an update in the morning.
- resolved Feb 14, 2023, 07:24 PM UTC
Total Expert has successfully resolved Monday’s issue with imports and email sends which resulted in unscheduled database maintenance and placing background processing on hold. The fix included the creation of a new activity streams table which substantially lowers any risk of recurrence. While investigating the issue, the team placed a proactive hold on processing, which resulted in queues for imports and emails for 24 hours. The hold on processing was lifted Monday evening and the queues were successfully processed in sequential order. We are continuing to process activity logs and tasks; this work could range from the next few hours up to two days. Please note that Contact list updates may be delayed until we complete processing the events backlog. Our team is tightly managing this final stage to mitigate impact on the overall performance of the platform. We apologize for the disruption and thank you for your patience as we worked through the issue. Our team will continue to post updates to the Total Expert status page.