Increase in errors at 1.0 and 2.0 kiosks

Update timeline

1. investigating Jun 14, 2024, 09:41 PM UTC

   We are investigating an issue affecting a limited number of 1.0 and 2.0 stations across several cities this week. Affected stations are falling into an unresponsive state on Admin and returning INI_META, INI_CNCT, or PIF_001 errors on restart. In many cases, the issue can be resolved by unpairing the PC from the station and re-pairing it. Replacing the PC with a newly imaged PC works as an alternative. The team is monitoring the issue and investigating the root cause of these kiosk issues. We will provide updates when more details are available.

2. investigating Jun 17, 2024, 09:00 PM UTC

   We are continuing to see this issue crop up sporadically at 1.0 and 2.0 stations and we are investigating the root cause. For similar cases, please follow the steps outlined here and submit a ticket with the station details for visibility.

3. investigating Jun 21, 2024, 09:39 PM UTC

   Our team is continuing to investigate this issue at 1.0 and 2.0 kiosks. We are zeroing in on a root cause. While the investigation proceeds, we recommend that operations teams visit kiosks that go unresponsive and, if the kiosks return one of the INI_META, INI_CNCT, or PIF_001 errors on restart, attempt to unpair and re-pair the PC to the kiosk. We look forward to providing more insights and another update next week.

4. investigating Jun 26, 2024, 01:57 PM UTC

   We are continuing to investigate this issue. Thank you for your patience as we work to find a resolution.

5. investigating Jun 28, 2024, 09:57 PM UTC

   We are continuing to explore options to resolve this issue. We apologize for the ongoing inconvenience of unanticipated kiosk maintenance. Thank you for your patience. We look forward to providing more updates next week.

6. investigating Jul 01, 2024, 10:56 PM UTC

   Thank you for your patience and collaboration as we work through this issue. With a busy holiday weekend on the horizon, we want to provide additional information and an option for mitigating the impact in your city. Issue Details: 1.0 and 2.0 kiosks must provide a token when they communicate with the BCycle system. Every 6 months, the system provides the kiosks with a new token and, in recent weeks, many kiosks have failed to accept their new token when the system issued one. Without the new token, these kiosks lose access to the BCycle system and fall into an unresponsive state. Work In Progress: Our team is reviewing the kiosk client code, looking for the reason for the failed acceptance of the new token so that we can develop a fix. Mitigation Option: Until a fix is completed in the code, operators can take steps to mitigate the issue. Kiosks will receive a new token when they are unpaired and re-paired. You can get ahead of the issue by proactively re-pairing any critical kiosks. For example, if a station is expected to be highly utilized on July 4th, you can take this step there today. Please see the BCycle Help Center for more details and/or to connect with our team.

7. identified Jul 09, 2024, 10:06 PM UTC

   Thank you again for your patience as we work through this issue at kiosks. Over the next several days, in coordination with operators, our team will be applying updates remotely to kiosks to address this issue and ensure that the kiosks reliably maintain their connection to the BCycle system. We will reach out to operators with schedule details.

8. monitoring Jul 12, 2024, 06:50 PM UTC

   This afternoon, our team is running updates for all kiosks in the DEN tenant (everywhere except Philadelphia and LA). The update extends the token expiration date at the kiosks. No downtime is expected, but kiosks may briefly report a 'cannot retrieve subscription types' error. This update will resolve this immediate issue. Further work to address the root cause is planned for the future.

9. resolved Jul 12, 2024, 09:34 PM UTC

   Kiosk updates have been completed and this incident has been resolved. Thank you!