Loops had 13 outages in the last 2 years totaling 33h 11m of downtime — averaging 0.5 incidents per month.
There were 13 Loops outages since September 16, 2024 totaling 33h 11m of downtime. Each is summarised below — incident details, duration, and resolution information.
Email metrics may be slow to appear or not appear on refresh. We are currently investigating the issue.
We're continuing to make progress processing a backlog of email metrics.
The metrics processing pipeline has caught up. No metrics were lost, this was only a delay in processing.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently investigating this issue.
We are currently working through the backlog of webhook messages.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating issues with delivery of outbound webhooks
This incident has been resolved.
Between approximately 13:42 and 14:18 UTC attempted webhook deliveries failed while our webhooks database was unavailable due to an unplanned outage. Webhooks messages during that period were lost and will not be delivered. We’ve prioritized work to make webhooks more resilient to this sort of failure. Email message sending was not impacted during this event.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Between Sept 3 03:40 UTC and Sept 3 08:30 UTC Loops experienced a period of elevated error rates where API requests intermittently returned 500’s due to high CPU use on our primary database. We resolved the issue and additionally made monitoring and process changes to speed responses in the future.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results. Email queues will be processed shortly for Loops and Campaigns.
This incident has been resolved.
Marketing email sending queues are delayed in processing their payloads. We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Campaigns, loops, and their preview emails are successfully being added to our worker queues, but there is a delay in processing the jobs. Contact updates and transactional email endpoints unlikely to be impacted. We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating an issue with an upstream provider, we're working to mitigate the impact.
We're seeing signs of recovery and continuing to monitor while we wait for an "all-clear" from our upstream provider.
All good!
We are currently investigating this issue.
The issue is resolved and we are monitoring.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
This incident has been resolved.
A misconfiguration by our database vendor prevented our database from restarting successfully. The misconfiguration was identified and remedied. Campaigns scheduled to send during the outage are now sent. Loop emails scheduled to send during the outage are now sent. Transactional emails or API calls made during the outage window need to be retried.