Prismatic incident

Increased error rates in execution pipeline in us-east-2 region

Minor Resolved View vendor source →

Prismatic experienced a minor incident on December 27, 2022 affecting Integration Runner, lasting 2h 39m. The incident has been resolved; the full update timeline is below.

Started
Dec 27, 2022, 05:44 PM UTC
Resolved
Dec 27, 2022, 08:23 PM UTC
Duration
2h 39m
Detected by Pingoru
Dec 27, 2022, 05:44 PM UTC

Affected components

Integration Runner

Update timeline

  1. investigating Dec 27, 2022, 05:44 PM UTC

    We are currently investigating this issue.

  2. investigating Dec 27, 2022, 06:19 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Dec 27, 2022, 06:40 PM UTC

    We have opened an issue with AWS support and error rates appear to be substantially lower at this time. We continue to monitor the issue.

  4. monitoring Dec 27, 2022, 07:02 PM UTC

    AWS has confirmed they are investigating increased network latency for S3 buckets in the us-east-2 region. The Prismatic team suspects this is a continuation of the networking problems AWS experienced late last week, but that is currently only speculation. We will continue to monitor this issue, work with AWS to provide any helpful information we can, and keep this page updated with the latest developments. Thank you for your patience.

  5. monitoring Dec 27, 2022, 07:29 PM UTC

    AWS has confirmed that they believe the underlying issue has been addressed and that service has been restored to normal operation. We have seen Prismatic services restored to normal operation as a result, but we will continue to monitor things for a period of time before we resolve this issue.

  6. resolved Dec 27, 2022, 08:23 PM UTC

    This incident has been resolved.