LeadSimple experienced a major incident on July 8, 2023 affecting Email Delivery and Outbound SMS and 1 more component, lasting 1h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 08, 2023, 04:20 PM UTC
The third-party vendor responsible for hosting our background job database is performing unscheduled maintenance on our database. This is interfering with LeadSimple's ability to perform any tasks in the background, including sending emails and text messages. These jobs will be performed inconsistently until the downtime resolves. We are investigating possible solutions.
- monitoring Jul 08, 2023, 05:04 PM UTC
Our database provider appears to be back up. We are monitoring for any continuing effects.
- resolved Jul 08, 2023, 11:52 PM UTC
This incident has been resolved and we are properly processing background tasks again. We are still evaluating if any background task was lost or not. We'll be updating this accordingly.
- postmortem Jul 12, 2023, 03:15 PM UTC
On July 8th, around 9:20 PDT we detected that some of our attempts to connect with our Redis instance - hosted by OpenRedis - were failing. We use Redis to schedule background jobs such as automatic emails, automatic text messages, Autopilot, integration with third-party PMS, email syncing, etc. The outage was intermittent, and it caused roughly ~5% of our jobs to fail. However, all of these jobs were correctly rescheduled and ran at a later time once the outage was solved, which means there was no data loss.