Pinata incident

Degraded Pinning and Gateway Performance

Minor Resolved View vendor source →

Pinata experienced a minor incident on August 15, 2023 affecting API and Dedicated Gateways, lasting 5h 42m. The incident has been resolved; the full update timeline is below.

Started
Aug 15, 2023, 11:50 PM UTC
Resolved
Aug 16, 2023, 05:32 AM UTC
Duration
5h 42m
Detected by Pingoru
Aug 15, 2023, 11:50 PM UTC

Affected components

APIDedicated Gateways

Update timeline

  1. identified Aug 15, 2023, 11:50 PM UTC

    We are currently experiencing some intermittent performance issues with pinning files to IPFS and Gateway fetching. This is mainly due to a memory leak in the ipfs-kubo repo that has been reported to Protocol Labs. We are doing what we can with our own infrastructure to mitigate the issues but there is still some affects at the moment.

  2. identified Aug 16, 2023, 02:43 AM UTC

    We have temporarily rerouted traffic which has mostly solved the degradation, however speeds are still slower than they should be. We are working on the original routing to bring those speeds back to 100% expected performance.

  3. identified Aug 16, 2023, 04:50 AM UTC

    We have successfully implemented a fix for the us region, and have begin to migrate traffic back and monitor traffic while doing so.

  4. monitoring Aug 16, 2023, 04:51 AM UTC

    Monitoring of the fix is showing signs of stability in all regions.

  5. resolved Aug 16, 2023, 05:32 AM UTC

    Incident appears to have been fully resolved