Rival incident

Windows Service Outage

Critical Resolved View vendor source →

Rival experienced a critical incident on July 19, 2024 affecting Recruiting UI and Workflow UI and 1 more component, lasting 8h 46m. The incident has been resolved; the full update timeline is below.

Started
Jul 19, 2024, 07:38 PM UTC
Resolved
Jul 20, 2024, 04:25 AM UTC
Duration
8h 46m
Detected by Pingoru
Jul 19, 2024, 07:38 PM UTC

Affected components

Recruiting UIWorkflow UILearning UIPerformance - Agile UIAuthAzure East-2Performance - Wingspan UICX UIOAuth APIePosting UI

Update timeline

  1. investigating Jul 19, 2024, 01:18 PM UTC

    All windows systems received a faulty security update from a third-party update. This appears to be affecting companies around the world. Our team is focused on recovery and working to restore services.

  2. identified Jul 19, 2024, 03:48 PM UTC

    Our team continues to work through the recovery steps.

  3. identified Jul 19, 2024, 03:56 PM UTC

    We are continuing to work on a fix for this issue.

  4. identified Jul 19, 2024, 04:28 PM UTC

    All International services hosted in our CA1 data center have been restored. Work continues on the remaining US-based (US1) services.

  5. identified Jul 19, 2024, 05:24 PM UTC

    Work continues on the remaining US-based (US1) services.

  6. identified Jul 19, 2024, 07:38 PM UTC

    The Recruit application and US1 SFTP service have been restored to operational state. Work continues on the remaining US-based (US1) services.

  7. identified Jul 19, 2024, 08:54 PM UTC

    Work continues on the remaining US-based (US1) services. The team is working on bringing up core systems that the applications rely on. Some instances of applications will start coming online shortly.

  8. identified Jul 19, 2024, 09:05 PM UTC

    The Learn application, Perform (Wingspan) application and Recruit microsites remain offline but will be restored shortly.

  9. identified Jul 19, 2024, 11:32 PM UTC

    The Learn application and Perform (Wingspan) application have been brought online with reduced capacity. Some Recruit microsites (ePosting/iPosting) remain offline. The team continues to work toward full production restoration.

  10. identified Jul 20, 2024, 03:14 AM UTC

    All services have been restored to functionality. Our team will continue to monitor the systems.

  11. identified Jul 20, 2024, 03:15 AM UTC

    All services have been restored to functionality. Our team will continue to monitor the systems.

  12. monitoring Jul 20, 2024, 03:15 AM UTC

    All services have been restored to functionality. Our team will continue to monitor the systems.

  13. resolved Jul 20, 2024, 04:25 AM UTC

    All systems have are running normally.