Storj incident

Issue with edge services' outage affecting customers in Europe and North America

Critical Resolved View vendor source →

Storj experienced a critical incident on November 8, 2024 affecting US1 - API and EU1 - Linksharing and 1 more component, lasting 4h 23m. The incident has been resolved; the full update timeline is below.

Started
Nov 08, 2024, 07:10 PM UTC
Resolved
Nov 08, 2024, 11:34 PM UTC
Duration
4h 23m
Detected by Pingoru
Nov 08, 2024, 07:10 PM UTC

Affected components

US1 - APIEU1 - LinksharingEU1 - GatewayUS1 - LinksharingUS1 - Gateway

Update timeline

  1. investigating Nov 08, 2024, 02:43 PM UTC

    We are experiencing an issue with edge services' outage affecting customers in the Europe and North America regions. Our engineering team is actively investigating. We apologize to all who are affected by the disruption.

  2. identified Nov 08, 2024, 03:38 PM UTC

    The issue has been identified and a fix is being implemented. The team will send another update in 15 minutes.

  3. identified Nov 08, 2024, 03:58 PM UTC

    We've implemented a fix and gateway services (S3 API) is gradually coming back online. Linksharing is still experiencing an issue.

  4. identified Nov 08, 2024, 04:20 PM UTC

    We are continuing to bring gateway services (S3 API) back online, requests in U.S. are seeing error rates decrease. E.U. gateway services are coming back online.

  5. identified Nov 08, 2024, 04:30 PM UTC

    U.S. Gateway (S3 API) services are restored. E.U. Gateway services are seeing decreased error rates and will be restored soon.

  6. identified Nov 08, 2024, 04:35 PM UTC

    E.U. Gateway services (S3 API) are restored. The team is focusing on bringing back online linksharing next.

  7. identified Nov 08, 2024, 05:08 PM UTC

    Gateway services (S3 API) are still online and functioning normally. Linksharing is being brought back online.

  8. identified Nov 08, 2024, 05:15 PM UTC

    Linksharing is being brought online in the U.S. and error rates are dropping. E.U. will be restored next.

  9. monitoring Nov 08, 2024, 05:29 PM UTC

    All services are operational. We are continuing to monitor.

  10. monitoring Nov 08, 2024, 05:44 PM UTC

    US1 API is now experiencing errors affecting all US1 services (gateway, linksharing, uplink). EU1 and AP1 are unaffected.

  11. monitoring Nov 08, 2024, 06:18 PM UTC

    US1 API has been restored, we are monitoring linksharing and gateway but error rates are decreasing. EU1 and AP1 continue to function normally.

  12. monitoring Nov 08, 2024, 06:44 PM UTC

    US1 API services are operational. We are still monitoring US1 linksharing and gateway elevated error rates. EU1 Linksharing and EU1 gateway are also now seeing elevated error rates.

  13. monitoring Nov 08, 2024, 07:10 PM UTC

    We are deploying an additional fix. All services are stabilizing, but the situation is ongoing so we continue to monitor and fix as we see issues.

  14. monitoring Nov 08, 2024, 08:03 PM UTC

    All services have been restored for nearly all customers. We are currently addressing isolated cases with individual customers to ensure full resolution. We continue to monitor.

  15. resolved Nov 08, 2024, 11:34 PM UTC

    After extended monitoring, we are marking this incident as resolved. We again apologize to all who are affected by the disruption and appreciate your patience and support. The team will conduct a post mortem and we share more details as we investigate fully.