Factbird incident

Stops not ending correctly

Major Resolved View vendor source →

Factbird experienced a major incident on October 12, 2022 affecting Data Analysis, lasting 15d 2h. The incident has been resolved; the full update timeline is below.

Started
Oct 12, 2022, 06:24 AM UTC
Resolved
Oct 27, 2022, 09:05 AM UTC
Duration
15d 2h
Detected by Pingoru
Oct 12, 2022, 06:24 AM UTC

Affected components

Data Analysis

Update timeline

  1. monitoring Oct 12, 2022, 06:24 AM UTC

    After a deployment of our stops finding service yesterday, some customers are experiencing issues with ongoing stops across production data where there should be no stops. We have rolled back to the previous stops service yesterday to mitigate the impact, and are now working on recovering the correct stops in the period. If you encounter ongoing stops that looks wrong, please email us at [email protected], so we can get your line back to operational status as quickly as possible.

  2. monitoring Oct 17, 2022, 11:48 AM UTC

    We are aware that the problem has occurred for some customers. We are working hard to resolve this problem with all hands on deck.

  3. monitoring Oct 17, 2022, 02:40 PM UTC

    We have resolved the issue for all customers now. We are still working on recalculate stop durations within the time of the incident to validate stop durations are correct. We expect to recover stop durations with no data loss. We will keep updating on the progress.

  4. resolved Oct 27, 2022, 09:05 AM UTC

    This incident has been resolved.