Zipline incident

High latency on web application

Major Resolved View vendor source →

Zipline experienced a major incident on March 20, 2023 affecting Core Platform and Login/SSO and 1 more component, lasting 1h 24m. The incident has been resolved; the full update timeline is below.

Started
Mar 20, 2023, 05:44 PM UTC
Resolved
Mar 20, 2023, 07:08 PM UTC
Duration
1h 24m
Detected by Pingoru
Mar 20, 2023, 05:44 PM UTC

Affected components

Core PlatformLogin/SSOGraphQL APISearchPublishingNotificationsFilesInsights

Update timeline

  1. investigating Mar 20, 2023, 05:44 PM UTC

    We are seeing high latency on page requests on the Zipline web application. Our engineers are currently investigating.

  2. investigating Mar 20, 2023, 06:11 PM UTC

    Response rates have come back down to typical rates. We have temporarily disabled Resource Library for a handful of customers while we further investigate

  3. monitoring Mar 20, 2023, 06:30 PM UTC

    Resource Library has been re-enabled for a majority of customers. Out of an abundance of caution, we are re-enabling customers one by one and monitoring latency to ensure we do not trigger more latency.

  4. monitoring Mar 20, 2023, 06:42 PM UTC

    Resource Library has been restored for all customers. We are now monitoring for any changes to latency.

  5. resolved Mar 20, 2023, 07:08 PM UTC

    Latency rates have remained stable. The incident is resolved.