Gitpod incident

Dashboard loads slowly and users cannot start workspaces

Major Resolved View vendor source →

Gitpod experienced a major incident on May 2, 2024 affecting [PAYG] Dashboard and [PAYG] Workspaces, lasting 2h 29m. The incident has been resolved; the full update timeline is below.

Started
May 02, 2024, 05:44 PM UTC
Resolved
May 02, 2024, 08:13 PM UTC
Duration
2h 29m
Detected by Pingoru
May 02, 2024, 05:44 PM UTC

Affected components

[PAYG] Dashboard[PAYG] Workspaces

Update timeline

  1. investigating May 02, 2024, 05:44 PM UTC

    We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

  2. investigating May 02, 2024, 05:58 PM UTC

    We've applied mitigation, this allows for workspace starts. However, the API request and error rate is still elevated, which is causing the dashboard and workspace start experience to be slow. We're still investigating.

  3. monitoring May 02, 2024, 07:52 PM UTC

    We've restored the system, workspace starts and the dashboard experience are normal again. We had to rollback a change that was putting excessive load on the system, it was causing API request duration to be too high.

  4. resolved May 02, 2024, 08:13 PM UTC

    We are pleased to confirm that the issue has been successfully resolved, and all system functionalities have been restored to normal operation.