T2O USA Incident: A subset of Live Invitation Emails are not being sent to Interviewers

Update timeline

1. identified Mar 28, 2025, 06:55 PM UTC

   We are experiencing a service disruption with Sending Live Invitation Emails to Interviewers within our T2O US environment. A subset of Live Invitation Emails to Interviewers are not being sent for upcoming interviews. Candidates are unaffected. We are making every effort to reduce the impact and length of the disruption. We apologize for any inconvenience this may cause, and we thank you for your understanding. We have identified the root cause and plan to release a hot fix. We'll continue to make updates as we progress. Individual system components are listed below.

2. identified Mar 28, 2025, 07:59 PM UTC

   We are continuing to work to release the hot fix and will you keep you updated as we progress.

3. identified Mar 28, 2025, 09:04 PM UTC

   We are continuing to work to release the hot fix. We are actively validating the resolution in lower environments. We will you keep you updated as we progress.

4. identified Mar 28, 2025, 09:59 PM UTC

   We are continuing to work to release the hot fix. We have successfully validated the resolution in lower environments. We are actively pushing the resolution to prod environments. We will you keep you updated as we progress.

5. monitoring Mar 28, 2025, 10:33 PM UTC

   A fix has been implemented to all T2O prod environments and we are monitoring the results. Our next step is resend impacted Live interviewer emails. We will keep you updated as we continue.

6. resolved Mar 28, 2025, 11:41 PM UTC

   This incident has been resolved. We are working resend impacted Live interviewer emails. We will update once the re-triggering has started. Final edit: We started reprocessing the messages and they were queued for delivery as of 7pm MDT. No additional updates are planned for this incident.