Flowhub experienced a major incident on August 16, 2022, lasting 5h 53m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Aug 16, 2022, 09:31 PM UTC
A feature in Flowhub Classic that helps scale with demand is taking longer to scale than expected. We are aware of the issue and are working to resolve right now. We expect this to be resolved shortly.
- identified Aug 16, 2022, 11:03 PM UTC
We have successfully added some resources to POS and are seeing some improvement in performance. We're continuing to add more resources to the deployment, and are expecting a full resolution within the next 15-30 minutes.
- investigating Aug 17, 2022, 12:02 AM UTC
After scaling up more resources, we're noticing that folks using Flowhub Classic are not getting redirected over to using the new resources. As such, the performance is still degraded for the end user and we're working with our vendor and Google to help troubleshoot this issue.
- investigating Aug 17, 2022, 02:25 AM UTC
We are on a call with Google to try to diagnose the issue with our new servers. We're continuing to have a degraded platform that can service only so many requests at a time without the ability to scale up and service more requests. ETA is still open, but we hope to have an idea soon.
- resolved Aug 17, 2022, 03:25 AM UTC
After working with Google support, we determined that the issue was actually related to MongoDB refusing connections to Flowhub Classic. We restarted MongoDB, and it came up cleanly. Flowhub Classic is performing normally at this time, and we will follow up with a post-mortem to better understand the root cause of the issues today. We will also continue to monitor through the night to ensure Flowhub Classic continues to run properly.