Semaphore incident

Slow job processing on e2 and f1 machine types

Major Resolved View vendor source →

Semaphore experienced a major incident on April 10, 2025 affecting Semaphore, lasting 1h 40m. The incident has been resolved; the full update timeline is below.

Started
Apr 10, 2025, 09:55 AM UTC
Resolved
Apr 10, 2025, 11:36 AM UTC
Duration
1h 40m
Detected by Pingoru
Apr 10, 2025, 09:55 AM UTC

Affected components

Semaphore

Update timeline

  1. investigating Apr 10, 2025, 09:42 AM UTC

    We are currently investigating this issue.

  2. identified Apr 10, 2025, 09:46 AM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Apr 10, 2025, 09:55 AM UTC

    We are continuing to work on a fix for this issue.

  4. identified Apr 10, 2025, 10:12 AM UTC

    Job delays on the Ubuntu Noble (24.04) image.

  5. identified Apr 10, 2025, 11:05 AM UTC

    A fix was deployed for e2 machines. We are continuing work on f1 machines.

  6. monitoring Apr 10, 2025, 11:16 AM UTC

    We have deployed a fix for f1 machines, and we are monitoring the results.

  7. resolved Apr 10, 2025, 11:36 AM UTC

    This incident has been resolved.