BCycle experienced a minor incident on September 17, 2023 affecting IoT Hub and Service Fabric and 1 more component, lasting 3d 14h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 17, 2023, 11:17 PM UTC
We are currently seeing degradation in message processing speeds again similar to our previous incident that occurred last week. As a response to this, we are reimplementing the temporary measure to make kiosks available for checkouts through the mobile app. Please read the following explanation for details: As a temporary measure, we are marking kiosks as Active if they are currently full-service kiosks that are visible to the public. This measure will support the rider in your systems as we continue to work to address the root cause of the kiosk message processing delays. As an operator, you can continue to monitor kiosk connectivity in the Kiosk & Station Maintenance grid in BCycle Admin; if you see the Sensor Reading Timestamp lagging behind the current 15-minute window, but advancing in time (i.e. it appears consistently, say, 45 minutes behind the current time), then that kiosk is affected by the delays. Kiosks that stop reporting for longer may in fact be disconnected and require a visit.
- identified Sep 18, 2023, 09:45 PM UTC
We are continuing to work on a fix for this issue. In the short term, kiosks will continue to be set as Active, regardless of kiosk sensor reading timestamps, and will therefore be available for riders for mobile app checkout. We expect to release additional logging and other improvements on Tuesday, following the completion of testing. Thank you for your patience as we work toward a final solution.
- identified Sep 19, 2023, 09:54 PM UTC
We are releasing improvements for kiosk heartbeat messaging this evening. We will monitor messages following the release. In the short term, kiosks will continue to be set as Active, regardless of kiosk sensor reading timestamps, and will therefore be available for riders for mobile app checkout.
- monitoring Sep 20, 2023, 04:13 PM UTC
A fix was implemented with last night's deployment and our monitoring shows that kiosk message processing is running at good speeds. At this time, we are turning the function that sets kiosks as active or unresponsive back on; now kiosk connection status will update automatically with kiosk heartbeat messages. We will continue to closely monitor the system with these changes in place.
- resolved Sep 21, 2023, 02:15 PM UTC
Monitoring continues to show positive results from the changes implemented this week and this incident has been resolved. Thank you for your patience as we worked on this issue.