BCycle experienced a minor incident on February 22, 2022 affecting Service Fabric, lasting 13h 59m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Feb 21, 2022, 06:47 PM UTC
Our Service Fabric for 3.0 Docks and New Client Kiosks is currently recovering, working through a back-log of messages. This may impact records/performance of New Client Kiosks and 3.0 docks - We do expect that things will recover on their own, but the system does need some time to catch-up to real-time events. We will update this status when we see that the system has fully recovered.
- investigating Feb 22, 2022, 04:03 PM UTC
We have detected continuous degraded performance in our Service Fabric solution which is affecting functionality in several systems. In order to help mitigate this issue, a full redeploy of our Service Fabric solution will be done at 11:30 AM CST. This will result in a 30 minute downtime to the BCycle system impacting new kiosk clients and 3.0 docks. *Update* We are performing some additional testing of the mitigation deployment, this is pushing back the full redeploy of our Service Fabric solution by an hour. This maintenance, resulting in a 30 minute downtime, will not occur until 12:30 PM CST. We apologize for this inconvenience, but we want to get this right.
- investigating Feb 22, 2022, 06:45 PM UTC
We have detected continuous degraded performance in our Service Fabric solution which is affecting functionality in several systems. In order to help mitigate this issue, a full redeploy of our Service Fabric solution will be done at 11:30 AM CST. This will result in a 30 minute downtime to the BCycle system impacting new kiosk clients and 3.0 docks. *Update* We are performing some additional testing of the mitigation deployment, this is pushing back the full redeploy of our Service Fabric solution by an hour. This maintenance, resulting in a 30 minute downtime, will not occur until 12:30 PM CST. We apologize for this inconvenience, but we want to get this right. *Update* We are now deploying at 1:00 PM CST.
- investigating Feb 22, 2022, 07:33 PM UTC
We are still troubleshooting in our test environment, we will update here when we are ready to begin the deployment.
- investigating Feb 22, 2022, 10:57 PM UTC
We are still working on our deployment, but it's going to take some more time. We will continue to work on this into the evening and make every attempt to get our changes pushed this evening.
- identified Feb 23, 2022, 02:28 AM UTC
We will begin deployment at 8:45PM. Kiosks running the new kiosk client and 3.0 docks will be unavailable.
- monitoring Feb 23, 2022, 04:12 AM UTC
Deployment is complete. We will be monitoring into the evening.
- resolved Feb 23, 2022, 06:02 AM UTC
System is back online and operating as expected. We will continue to monitor it closely over the coming days. Thank you for your patience.