Fulfil.IO incident

Shipping label generation is experiencing issues related to Google Cloud networking issues

Minor Resolved View vendor source →

Fulfil.IO experienced a minor incident on March 27, 2020 affecting WMS and PDF Generator, lasting 1h 16m. The incident has been resolved; the full update timeline is below.

Started
Mar 27, 2020, 08:08 PM UTC
Resolved
Mar 27, 2020, 09:25 PM UTC
Duration
1h 16m
Detected by Pingoru
Mar 27, 2020, 08:08 PM UTC

Affected components

WMSPDF Generator

Update timeline

  1. investigating Mar 27, 2020, 08:08 PM UTC

    We are investigating an issue related to Amazon S3 preventing S3 related operations like label generation experiencing delays or partial outage.

  2. identified Mar 27, 2020, 08:30 PM UTC

    Google cloud has updated their status to indicate they are experiencing connectivity issues with other cloud providers. https://status.cloud.google.com//incident/cloud-networking/20004?_ga=2.268214414.-1688214264.1573732471

  3. monitoring Mar 27, 2020, 08:39 PM UTC

    The issue seems to be resolved at this point, though google cloud engineers are still investigating the increased latency they are observing between GCP and other cloud services.

  4. resolved Mar 27, 2020, 09:25 PM UTC

    Google has marked the issue as resolved. We are seeing low latencies and we have not seen any timeouts. We are marking this issue as resolved