Fulfil.IO incident
Investigating reports of PDF generation being slow
Fulfil.IO experienced a minor incident on June 21, 2021 affecting PDF Generator, lasting 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 21, 2021, 07:30 PM UTC
We are currently investigating this issue.
- identified Jun 21, 2021, 07:38 PM UTC
The PDF conversion for all reports (except shipping labels) is affected. We are working on a solution.
- identified Jun 21, 2021, 07:50 PM UTC
Google cloud has updated that: "Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions." This is affecting the autoscaled PDF conversion system. We are closely following the status updated from google at https://status.cloud.google.com/ In the meantime, we are working on setting up a fallback for PDF conversion. PS: No other services are impacted at this time
- monitoring Jun 21, 2021, 08:03 PM UTC
The fallback PDF conversion services have now kicked in. PDF generation is back to normal. We are closely monitoring the service.
- resolved Jun 21, 2021, 08:26 PM UTC
The incident has been resolved. PDF conversion is working normally. The Google cloud issue is ongoing, but the failover PDF conversion process is working as expected in the meantime. Recovery of the standard conversion service will be done only after the Google cloud issues have been resolved. The switchover is expected to be done post business hours and will not affect service availability.