BCycle incident

BCycle System Network Outage

Critical Resolved View vendor source →

BCycle experienced a critical incident on July 19, 2024 affecting DEN - BCycle Admin and DEN - Portal API and 1 more component, lasting 14h 47m. The incident has been resolved; the full update timeline is below.

Started
Jul 19, 2024, 05:18 AM UTC
Resolved
Jul 19, 2024, 08:06 PM UTC
Duration
14h 47m
Detected by Pingoru
Jul 19, 2024, 05:18 AM UTC

Affected components

DEN - BCycle AdminDEN - Portal APIIoT HubPHL - BCycle AdminPHL - Portal APIService FabricLAX - BCycle AdminStream AnalyticsLAX - Portal APIProgram Websites

Update timeline

  1. investigating Jul 19, 2024, 05:18 AM UTC

    We are currently investigating an issue with networking failures in our system. We will continue to update on the impact of this in our system as we know more.

  2. identified Jul 19, 2024, 06:08 AM UTC

    We have identified root cause with a network failure. We are working to resolve the issue and will continue to update with more information when possible.

  3. identified Jul 19, 2024, 07:39 AM UTC

    We are continuing to work on a fix for this issue.

  4. identified Jul 19, 2024, 11:06 AM UTC

    A fix is currently being rolled out to affected components across our system. We appreciate your patience as we implement this solution.

  5. identified Jul 19, 2024, 11:08 AM UTC

    BCycle continues to actively investigate the systemwide outage, which is impacting trips, Admin, the BCycle App, BCycle.com. This outage is directly related to the Global Crowdstrike Outage, which is impacting global services, including major outages with airlines and emergency support services. BCycle has been actively working on applying a fix to our servers. Once that's done, path to full recovery is unclear at this time, but we will keep things updated here as we get closer to being fully back online. We apologize for this major disruption, and are doing everything we can to restore service as soon as possible. In the meantime, we advise you to communicate as needed to your local bike share team and riders.

  6. identified Jul 19, 2024, 12:55 PM UTC

    We are continuing to apply the fix to our servers. We expect services to begin to recover as all servers receive the fix. Sitefinity-based websites are available again. We will provide another update within the next hour.

  7. monitoring Jul 19, 2024, 01:52 PM UTC

    The fix has been implemented. We are seeing systems recover and user trips are occurring in all three tenants. We will continue monitoring throughout the morning to ensure a complete return to normal service. Please reach out via the BCycle Help Center with any open issues. Thank you!

  8. resolved Jul 19, 2024, 08:06 PM UTC

    This incident has been resolved and all systems are running normally.