Pulumi incident

Pulumi Cloud: Degraded performance

Major Resolved View vendor source →

Pulumi experienced a major incident on September 18, 2023 affecting AWS ec2-us-west-2 and Deployments, lasting 10h 22m. The incident has been resolved; the full update timeline is below.

Started
Sep 18, 2023, 06:24 PM UTC
Resolved
Sep 19, 2023, 04:46 AM UTC
Duration
10h 22m
Detected by Pingoru
Sep 18, 2023, 06:24 PM UTC

Affected components

AWS ec2-us-west-2Deployments

Update timeline

  1. investigating Sep 18, 2023, 06:24 PM UTC

    Pulumi Deployments is experiencing degraded performance. We are investigating.

  2. investigating Sep 18, 2023, 06:48 PM UTC

    We are continuing to investigate this issue.

  3. investigating Sep 18, 2023, 06:49 PM UTC

    Pulumi Deployments and the API are experiencing degraded performance. We are continuing to investigate.

  4. investigating Sep 18, 2023, 06:55 PM UTC

    We are continuing to investigate this issue.

  5. investigating Sep 18, 2023, 06:57 PM UTC

    There is a partial AWS networking outage in multiple regions contributing to degraded API and Deployments availability for Pulumi Cloud.

  6. identified Sep 18, 2023, 08:34 PM UTC

    API degradation has been resolved. Pulumi Deployments continues to see degraded performance due to the underlying AWS networking outage. We are continuing to work on mitigation.

  7. identified Sep 19, 2023, 03:00 AM UTC

    Pulumi Deployments continues to see degraded performance due to an underlying AWS networking outage. We are working on finding alternative mitigations given the extended nature of the underlying issue.

  8. resolved Sep 19, 2023, 04:46 AM UTC

    The incident has been resolved.