Pix4D incident

Elevated errors on our backend infrastructure

Minor Resolved View vendor source →

Pix4D experienced a minor incident on December 7, 2021 affecting Processing infrastructure US and Processing infrastructure DE and 1 more component, lasting 17h 26m. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 03:50 PM UTC
Resolved
Dec 08, 2021, 09:17 AM UTC
Duration
17h 26m
Detected by Pingoru
Dec 07, 2021, 03:50 PM UTC

Affected components

Processing infrastructure USProcessing infrastructure DEProcessing infrastructure JPProcessing infrastructure KRPIX4Dengine Cloud API

Update timeline

  1. investigating Dec 07, 2021, 03:49 PM UTC

    We're experiencing an elevated level of errors on our backend due to issues from our cloud provider. Processing might be delayed or failing due to this underlying issue.

  2. identified Dec 07, 2021, 03:50 PM UTC

    The issue has been identified and we are monitoring our cloud provider services.

  3. monitoring Dec 08, 2021, 07:17 AM UTC

    The issues impacting our cloud provider seems mostly resolved and we are not observing any error anymore. We are putting back in the queue the failed processing jobs.

  4. monitoring Dec 08, 2021, 07:22 AM UTC

    All processing clusters are resuming normal operations. Our cloud provider has not yet closed the incident. We keep monitoring the situation while reprocessing all failed projects.

  5. resolved Dec 08, 2021, 09:17 AM UTC

    We are not observing any issues anymore in our services, processing is back to normal. All failed projects have been sent for reprocessing.