Mariana Tek experienced a notice incident on September 21, 2022 affecting Admin and Webhooks and 1 more component, lasting 6h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Sep 21, 2022, 05:03 PM UTC
Starting on 9/20/2022 9:00 ET, Mariana Tek’s tasking queues experienced delays with job processing. Automated tasks such as transactional emails, waitlist upgrades into class, penalty fees and webhooks were delayed. We can confirm that all membership payments processed as expected last night. At this time, studios will experience the following: Delay in Transactional emails: Delayed sending starting from 9pm EST last night (9/20) and continue to have a delay this morning. This does impact Livestream emails as well as class message feature in admin We strongly suggest manually emailing livestream emails from the studio’s email provider Waitlist spot holds & Penalty Fees processing: Spots may be sporadically holding longer than expected as the system processes the queue of customers that need to be automatically moved from the waitlist into class. Penalty fees may sporadically experience a longer than expected processing as well. We suggest monitoring your classes to manually move customers from the waitlist into the class to optimize studio operations at this time. Webhooks: Studios with custom integrations and Integration Partners may see delays in getting updated information via webhooks Our engineering team is actively reviewing and working on this with an urgent priority. Please reach out to the Mariana Tek Support team if you have any questions.
- monitoring Sep 21, 2022, 10:14 PM UTC
At 5:50 pm ET, we implemented an update to improve the speed for job processing. We are seeing a significant increase in the system’s ability to process automated jobs, especially transactional emails and webhooks. For transactional emails, customers may have questions about receiving old emails. We continue to monitor the system’s progress to process the automated jobs that have been queued today but have not been able to process yet. Please reach out to the Mariana Tek Support team if you have any questions.
- monitoring Sep 21, 2022, 11:51 PM UTC
We are continuing to monitor for any further issues.
- monitoring Sep 21, 2022, 11:54 PM UTC
We are continuing to monitor for any further issues.
- resolved Sep 22, 2022, 12:01 AM UTC
We are happy to announce that the job processing delay incident is now resolved. All delayed transaction emails since last night have now all been sent and all waitlist spot holds, penalty fee processing, and Webhooks are renewed to their normal state. The queues remain stable and continue to process jobs in a timely manner