Orbee experienced a critical incident on February 11, 2021 affecting Platform, lasting 2h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 11, 2021, 03:43 PM UTC
We are currently investigating issues with our API Gateway (access to our services) not being accessible and returning 503's.
- investigating Feb 11, 2021, 03:56 PM UTC
We are continuing to investigate the issue.
- investigating Feb 11, 2021, 04:19 PM UTC
We are continuing to investigate this issue, but have found a likely workaround that we are testing and deploying now.
- identified Feb 11, 2021, 05:04 PM UTC
We have identified the issue and are currently implementing a fix.
- monitoring Feb 11, 2021, 05:22 PM UTC
We have completed testing the fix and are monitoring the rollout now.
- monitoring Feb 11, 2021, 05:37 PM UTC
We are starting to see customers regain access to the platform. We will continue to monitor the situation for a while.
- resolved Feb 11, 2021, 06:06 PM UTC
The platform is back to normal and service access has resumed. We will continue to monitor throughout the morning. The root cause has been addressed. It relates to a third-party monitoring system that plugs into our services failing and caused our gateway systems to crash. We have updated this dependency so it cannot cause the failure of our gateways.