Pinata incident

Dedicated Gateways Outage

Critical Resolved View vendor source →

Pinata experienced a critical incident on November 2, 2023 affecting API and Dedicated Gateways and 1 more component, lasting 10h 8m. The incident has been resolved; the full update timeline is below.

Started
Nov 02, 2023, 03:01 PM UTC
Resolved
Nov 03, 2023, 01:09 AM UTC
Duration
10h 8m
Detected by Pingoru
Nov 02, 2023, 03:01 PM UTC

Affected components

APIDedicated GatewaysPublic GatewayWeb App

Update timeline

  1. identified Nov 02, 2023, 03:01 PM UTC

    Creation and retrieving lists of gateways is impacted. A major networking provider is down. We are looking into this issue.

  2. identified Nov 02, 2023, 05:21 PM UTC

    We are experiencing issues with user logins. We are currently working on fix for this issue.

  3. identified Nov 02, 2023, 05:30 PM UTC

    We are continuing to work on a fix for this issue.

  4. monitoring Nov 02, 2023, 06:08 PM UTC

    We are seeing gradual improvement to gateways. We are currently monitoring this issue.

  5. monitoring Nov 02, 2023, 06:10 PM UTC

    We are continuing to monitor for any further issues.

  6. monitoring Nov 02, 2023, 07:02 PM UTC

    Unfortunately, the network provider fixes are taking awhile to roll out. While some users are seeing improvement, others may still see impact. We will be continuing to monitor this issue.

  7. monitoring Nov 02, 2023, 11:37 PM UTC

    Dedicated gateway creation within Pinata is still affected by the Cloudflare outage affecting much of the internet today. Our team is monitoring the situation closely for update and will report back as soon as the situation changes.

  8. monitoring Nov 03, 2023, 12:20 AM UTC

    We’re beginning to see signs of recovery on our end, with many gateway creations / deletions occurring successfully. The Pinata team will continue to monitor before fully closing out this event.

  9. resolved Nov 03, 2023, 01:09 AM UTC

    Cloudflare has marked the systems that gateway creation relies on as operational, and our team can confirm this from our end as well. We’re marking things back to fully operational but will continue to monitor.