Stream incident

High error rate on Feed Realtime endpoint

Major Resolved View vendor source →

Stream experienced a major incident on January 6, 2021 affecting Personalization, lasting 10h 35m. The incident has been resolved; the full update timeline is below.

Started
Jan 06, 2021, 05:00 AM UTC
Resolved
Jan 06, 2021, 03:35 PM UTC
Duration
10h 35m
Detected by Pingoru
Jan 06, 2021, 05:00 AM UTC

Affected components

Personalization

Update timeline

  1. identified Jan 06, 2021, 08:20 AM UTC

    The issue has been identified and a fix is being implemented.

  2. monitoring Jan 06, 2021, 09:53 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Jan 06, 2021, 10:02 AM UTC

    Realtime updates for feeds are back to normal, we are still monitoring the traffic. The previous patch unfortunately did not resolve the problem and was causing realtime clients to retry the connection via the `Client not found, please reconnect` response.

  4. resolved Jan 06, 2021, 03:35 PM UTC

    This incident has been resolved.