Orbee experienced a critical incident on March 4, 2021 affecting Platform, lasting 6h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 04, 2021, 06:18 PM UTC
We are currently investigating this issue.
- investigating Mar 04, 2021, 06:57 PM UTC
We are continuing to investigate this issue.
- identified Mar 04, 2021, 07:32 PM UTC
We have identified a potential cause and are addressing it.
- identified Mar 04, 2021, 07:54 PM UTC
We have identified the cause to be a reporting database suffering hardware issues. We are isolating the reporting DB impact from other critical services and working to return all other services back online through our API Gateways.
- identified Mar 04, 2021, 08:16 PM UTC
We are deploying and monitoring our patches applied to our admin service. This outage only impacts access to the platform; it does not impact live products and services like emails, phone calls, and advertising.
- monitoring Mar 04, 2021, 08:24 PM UTC
We have isolated the issue and are monitoring production. Access to the platform should be back.
- monitoring Mar 04, 2021, 08:54 PM UTC
The platform is back online and stable. There are a couple of edit features in the platform related to Orbee Reporting (monthly source/medium costs) that are disabled while we complete our reporting database fixes. We will share an update when these are complete.
- monitoring Mar 04, 2021, 09:57 PM UTC
We are continuing to address minor concerns with isolated features. Platform access and usage is available to all customers and operations by using the Platform are operational.
- monitoring Mar 04, 2021, 11:50 PM UTC
We are addressing the final edge-case issues due to our isolation of the reporting DB from administrative systems. The platform continues to be operational and available; we should have the last edge-cases completed soon.
- resolved Mar 05, 2021, 01:04 AM UTC
This issue is now completely resolved.