imgix incident

Issues with imgix web tools

Notice Resolved View vendor source →

imgix experienced a notice incident on March 8, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
Mar 08, 2024, 10:58 PM UTC
Resolved
Feb 20, 2024, 05:00 PM UTC
Duration
Detected by Pingoru
Mar 08, 2024, 10:58 PM UTC

Update timeline

  1. resolved Mar 08, 2024, 10:58 PM UTC

    We are investigating issues with imgix administration tools. There are reports of problems with calling our imgix upload API. The rendering service is not affected.

  2. postmortem Mar 08, 2024, 10:58 PM UTC

    # Postmortem # **What happened?** On Feb 20, 2024, at 19:00 UTC, Uploads using the \`/api/v1/sources/upload/\` API endpoint experienced slowness and some timeouts during this time. The issue had been completely resolved by Feb 21, 2024, 20:00 UTC. # **How were customers impacted?** Between Feb 20, 2024, 19:00 UTC, and Feb 21, 2024, 20:00 UTC, some requests to our upload API experienced slow responses, and a subset of requests resulted in timeout errors. Uploads using the \`/api/v1/sources//upload-sessions/\` endpoints were unaffected; uploads using the Asset Manager UI were unaffected. # **What went wrong during the incident?** Two compounding issues caused the slowdown. The first is a service update creating an issue with our /api/v1/sources/upload/\` API endpoint. Secondly, a simultaneous and separate slowdown affected the cloud servers responsible for executing the upload actions. The service update issue increased the load on our upload function, which was already under strain due to the cloud servers experiencing a slowdown. These factors combined to cause the slow and timed-out responses. # **What will imgix do to prevent this in the future?** We have streamlined the upload process so that this cannot happen again.