Happo incident

Increased error rates for snap-request jobs

Major Resolved View vendor source →

Happo experienced a major incident on June 2, 2025 affecting API, lasting 1h 12m. The incident has been resolved; the full update timeline is below.

Started
Jun 02, 2025, 07:05 PM UTC
Resolved
Jun 02, 2025, 08:17 PM UTC
Duration
1h 12m
Detected by Pingoru
Jun 02, 2025, 07:05 PM UTC

Affected components

API

Update timeline

  1. investigating Jun 02, 2025, 07:05 PM UTC

    We are investigating an issue with snap-requests randomly failing with a `HTTP error 404: http://happo-view-service/api/snap-requests/:id/payload -- Not Found` error.

  2. resolved Jun 02, 2025, 08:17 PM UTC

    We have found the root cause of this issue now. A temporary fix is currently active and we are soon to deploy a proper fix. The issue was caused by a stale cache that caused snap-request jobs to not save properly.