BCycle experienced a minor incident on July 18, 2022 affecting Service Fabric, lasting 10d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 18, 2022, 07:45 PM UTC
BCycle is currently investigating an intermittent checkout performance issue that may result in failed bike releases and returns. We are fully-focused on finding a resolution and will post relevant updates on a daily basis. As part of this effort, we will be adding additional logs in Service Fabric tonight. No downtime is expected.
- investigating Jul 18, 2022, 11:27 PM UTC
We are continuing to investigate this issue.
- investigating Jul 19, 2022, 09:15 PM UTC
Tuesday Update: Our investigation continues to be in-process. We deployed a hot-fix last night to expand logging capabilities and give greater visibility to system messaging processes. Already this visibility has given insight as to where delays in communication are happening but additional observation time is needed to identify a resolution. We will keep you posted on progress moving forward.
- investigating Jul 20, 2022, 09:34 PM UTC
Wednesday Update: While we've seen some improvement in checkout performance since the weekend, investigation of the current issue remains our teams' highest priority and we are working urgently on a resolution. The logging we put in place Monday night has improved confidence that we are on the right investigation path and we expect to deploy additional logging capabilities in the next 24 hrs to give even greater visibility to the issue as we progress towards finding a resolution. We plan to move forward with our standard deployment tonight and will keep you updated on additional progress.
- investigating Jul 21, 2022, 04:18 AM UTC
Deployment Update: Our standard deployment is complete. We will update Thursday with additional progress.
- investigating Jul 21, 2022, 04:56 PM UTC
We are performing an update to Service Fabric at 3pm CDT today to further expand logging capabilities as part of the investigation of the current issue. No downtime is expected.
- investigating Jul 21, 2022, 09:26 PM UTC
Our testing is going very well on the additional logging that we intended to get out into production today, but there is still one more piece that needs further testing before we deploy. This remains our first priority and expect to make this deployment early in the day tomorrow. There is still no intended downtime with this update.
- investigating Jul 22, 2022, 05:00 PM UTC
Performing an update to Service Fabric to further expand logging capabilities as part of the investigation of the current issue. No downtime is expected.
- investigating Jul 22, 2022, 05:36 PM UTC
Service Fabric update completed.
- investigating Jul 22, 2022, 09:31 PM UTC
Friday & Weekend Update: New system logging is providing our software team with new insights into the messaging delays that are contributing to the recently observed checkout and return issues. Implementing the next changes will require additional work early next week and we will provide an update on a potential release as soon as possible. We continue to see rates of successful Check-outs/Check-ins across systems comparable to normal operation, even with the current intermittent issue. We will continue to closely monitor system performance over the weekend as we continue efforts to resolve the issue. Should you begin to see higher rates of failure or other disruptions please notify us by leaving a voicemail for our Emergency Support on-call team (1-800-615-8735 Option #2).
- identified Jul 25, 2022, 10:33 PM UTC
Monday Update: we have identified a resolution which we believe will mitigate the intermittent issues impacting checkouts and returns. We are currently targeting deploying this update at midnight CST Wednesday, July 27th. This planned maintenance is expected to impact system access for riders and operators for approximately 60-90 minutes, during which user checkouts and Admin access will likely be impacted. Given the continued, relatively stable checkout performance we are seeing across systems we are choosing to schedule the release of this resolution 48hrs+ out to give operators ample time to communicate the planned downtime to riders, staff and stakeholders.
- identified Jul 26, 2022, 07:25 PM UTC
I apologize for the confusion our previous messaging created. In an effort to further clarify the timing questions we have received and reduce confusion, we will be deploying at 11:59pm CST on Wednesday 7/27.
- identified Jul 27, 2022, 07:35 PM UTC
We are all set for the deployment tonight at 11:59pm CST. This planned maintenance is expected to impact system access for riders and operators for approximately 60-90 minutes, during which user checkouts and portions of Admin access will be impacted. We will update this thread again when we begin the deployment and with any additional updates. Thank you for your patience.
- identified Jul 27, 2022, 07:35 PM UTC
We are all set for the deployment tonight at 11:59pm CST. This planned maintenance is expected to impact system access for riders and operators for approximately 60-90 minutes, during which user checkouts and portions of Admin access will be impacted. We will update this thread again when we begin the deployment and with any additional updates. Thank you for your patience.
- identified Jul 28, 2022, 05:02 AM UTC
We are beginning deployment now, and expect services to be interrupted for 60-90 minutes. We will update here with additional information.
- monitoring Jul 28, 2022, 05:54 AM UTC
Deployment is complete. Everything went out very well, and initial observations are looking very positive. We will continue to monitor closely throughout Thursday and will provide an additional update by end of day.
- resolved Jul 29, 2022, 12:04 AM UTC
Our internal logging throughout the day has shown a sizeable improvement in the area that was previously showing degraded performance. Given that data, we are closing the incident here. Please follow up via normal support communication methods for any additional observations. Thank you for your patience and feedback.