BCycle incident

SMS Messaging Availability/Legacy Kiosk Communications

Minor Resolved View vendor source →

BCycle experienced a minor incident on August 30, 2023 affecting DEN - BCycle Admin and PHL - BCycle Admin and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.

Started
Aug 30, 2023, 05:01 PM UTC
Resolved
Aug 31, 2023, 05:39 PM UTC
Duration
1d
Detected by Pingoru
Aug 30, 2023, 05:01 PM UTC

Affected components

DEN - BCycle AdminPHL - BCycle AdminLAX - BCycle AdminTwilio SMS Long Code, North America

Update timeline

  1. investigating Aug 30, 2023, 05:01 PM UTC

    We're currently investigating reports of SMS messages for user enrollment and other services being delayed or not received. Our team is aware of the issue and actively investigating. We will update this page with more details and progress as we have new information to report. Thank you for your patience while we address these issues.

  2. identified Aug 30, 2023, 07:28 PM UTC

    To address failures in registrations across many systems observed today, we are completing a rollback of the hot fix deployed last night. We apologize for the disruptions this has caused. We plan to perform this rollback at 3:00 PM CDT today, where you can expect approximately 5 minutes of system down time (3:00-3:05 CDT) to address these issues.

  3. monitoring Aug 30, 2023, 08:24 PM UTC

    We have deployed our changes and are monitoring the system. Initial testing is producing positive results. We will continue to monitor to ensure that services return to full operation. Thank you for your understanding.

  4. investigating Aug 30, 2023, 09:48 PM UTC

    We are currently investigating an outage affecting classic kiosk stations. We will provide updates as soon as we have more information.

  5. identified Aug 30, 2023, 11:01 PM UTC

    We have increased capacity on a server affecting system performance. We are seeing kiosks slowly reconnect. We are continuing to investigate the issue and will be monitoring the system for the next several hours.

  6. identified Aug 31, 2023, 12:31 AM UTC

    We are continuing to stay engaged on the current situation. As an additional measure, we anticipate that our deployment scheduled for tonight may have a positive impact, so we are electing to move the deployment up to 9:30 PM CDT in an effort to improve the situation on the field. We will continue to monitor the affected stations and work towards a resolution.

  7. monitoring Aug 31, 2023, 04:12 AM UTC

    Following the hot fix deployment, kiosks are reconnecting. We will continue to monitor the system to ensure a return to normal operations.

  8. resolved Aug 31, 2023, 05:39 PM UTC

    This incident is resolved, based on system monitoring. If you observe ongoing issues, please submit a ticket through the BCycle Help Center.