Orbee experienced a minor incident on August 29, 2023 affecting Email Marketing, lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 29, 2023, 08:00 PM UTC
We are currently investigating issues related to heightened error rates for email marketing lists and campaigns for some customers.
- identified Aug 29, 2023, 08:03 PM UTC
We have identified the issue causing raised error rates for lists and campaigns as an increased load on the database. We are looking into what is causing the increased load and are identifying fixes to address it. In the meantime, we do have an extensive number of email marketing events being processed for reporting. To help ease the database load, we have paused the processing of real-time statistics. Statistics will update hourly during this time. Do note the status of an email campaign or list is verified by our ESP, so if a campaign has a status of "Sent", then it actually sent. The statistics should roll in during the following hourly processing.
- monitoring Aug 29, 2023, 10:00 PM UTC
Given the additional changes and updates made to the email marketing database, we are seeing a reduction in load on the database. Error rates have also dropped significantly. We will provide an update later today as we continue to monitor the situation.
- resolved Aug 30, 2023, 09:02 PM UTC
Issues related to email marketing campaign error rates have been resolved the system has stabilized completely. Scheduling campaigns should happen without issue.