Recart experienced a major incident on June 14, 2023 affecting Messaging and Tracking, lasting 1h 30m. The incident has been resolved; the full update timeline is below.
Affected components
MessagingTracking
Update timeline
- investigating Jun 14, 2023, 07:33 AM UTC
We identified a problem with our ghosting subsystem, it started around yesterday 11 PM.
- monitoring Jun 14, 2023, 07:50 AM UTC
No changes have been released to the affected service before the problem started to happen. We re-deployed the service and it started to operate properly. We are monitoring the results.
- monitoring Jun 14, 2023, 09:04 AM UTC
The AWS outage from yesterday caused problems in our system. A few services lost their permission to read from SQS queues. We restarted all the services to make sure every permission-related info is updated and working. All the services are working properly.
- resolved Jun 14, 2023, 09:04 AM UTC
This incident has been resolved.