Alloy incident

[Investigating] Delays in Application Queue Updates and Missing Evaluations

Major Resolved View vendor source →

Alloy experienced a major incident on June 16, 2025 affecting Production API and Customer Dashboard, lasting 3h 40m. The incident has been resolved; the full update timeline is below.

Started
Jun 16, 2025, 10:49 PM UTC
Resolved
Jun 17, 2025, 02:30 AM UTC
Duration
3h 40m
Detected by Pingoru
Jun 16, 2025, 10:49 PM UTC

Affected components

Production APICustomer Dashboard

Update timeline

  1. investigating Jun 16, 2025, 10:50 PM UTC

    We are currently investigating an issue where some applications remain in the review queue after being actioned and rerun evaluations are not appearing on the Evaluations page. Additionally, we are aware of a potential disruption where some customers are not seeing new evaluation records since approximately 4:40 PM CST. Our engineering team is actively investigating and we will provide updates as soon as more information becomes available.

  2. identified Jun 17, 2025, 01:00 AM UTC

    Entities, Evaluations, Journey Applications, and Investigations that were created or updated since 5:45PM ET won't have up-to-date results when searched for in the dashboard. We have a fix in the works—verifying that it'll resolve the issue.

  3. monitoring Jun 17, 2025, 02:03 AM UTC

    System is now recovering. We anticipate return to normal in 30min.

  4. resolved Jun 17, 2025, 02:30 AM UTC

    The system has recovered.