Prodly incident

Subset of web instances have failed requests

Major Resolved View vendor source →

Prodly experienced a major incident on September 30, 2022 affecting AppOps Deployment and AppOps, lasting 4h 9m. The incident has been resolved; the full update timeline is below.

Started
Sep 30, 2022, 09:31 PM UTC
Resolved
Oct 01, 2022, 01:40 AM UTC
Duration
4h 9m
Detected by Pingoru
Sep 30, 2022, 09:31 PM UTC

Affected components

AppOps DeploymentAppOps

Update timeline

  1. investigating Sep 30, 2022, 09:31 PM UTC

    Seeing 5xx failures. Instances are being restarted.

  2. identified Sep 30, 2022, 10:01 PM UTC

    One instance was down, restarting.

  3. monitoring Sep 30, 2022, 10:07 PM UTC

    Instance restarted, monitoring errors.

  4. identified Sep 30, 2022, 10:09 PM UTC

    The issue has been identified and a fix is being implemented.

  5. monitoring Sep 30, 2022, 10:49 PM UTC

    Unhealthy instance taken out of rotation. Trying to get it back up, but other instances allow deployments.

  6. resolved Oct 01, 2022, 01:40 AM UTC

    This incident has been resolved.