DoControl Outage History

We are currently experiencing a partial service disruption affecting the GraphQL (GQL) API. Some users may encounter intermittent failures or delays when attempting to get responses related to permissions and settings when making GraphQL requests. Our engineering team is actively investigating the issue and working to restore full service as quickly as possible. We will continue to provide updates as more information becomes available. Thank you for your patience.

We are experiencing a critical system outage due to an Clickhouse.com infrastructure issue partially affecting google, our alerting system and dot. Clickhouse.com is currently experiencing an outage in our region. The issue is described in clickhouse.com status page. https://statuspage.incident.io/clickhousecloud/incidents/01KG9R4QYASZW9VXHG3S2YD4WD Current state: Google events, remediation and workflows will keep working. Impacted functionality: Google assets inventory page and data processing - not available Alerts generation (legacy + ai alerts) - not available Dot interaction with events and identities Our disaster recovery processes will able to reach 100% coverage and 0% data loss once Clickhouse is back on track. Our team is actively monitoring the situation and will restore service once Clickhouse resolves the underlying issue.

Clickhouse has deployed a fix resulting Google ingestion flow is back on track, And alerts processing is currently running to complete the backfill

Events from OneDrive, SharePoint, Teams, and Microsoft Entra are currently unavailable due to an ongoing Microsoft outage. We are waiting for Microsoft to resolve this issue on their end. Once restored, we will update this status and event activity will resume as normal.

We are experiencing a critical system outage affecting all services. Our team is actively investigating the issue and working to restore service as quickly as possible. We will provide updates as more information becomes available.

Over the past 2 hours, we have been experiencing issues with ingesting google drive asset updates, loading the Google drive asset Inventory screen, and performing bulk remediations. The cause is an ongoing incident affecting the database service we use. Workflow event-based executions are operating normally. Once solved, all updates made during this period will be processed, and no data will be lost. Our team is actively monitoring the situation and will provide further updates as new information becomes available.

We have identified that the issue is related to an incident reported by our database provider, which is experiencing failures when establishing connections to new pods in the us-east-1 region. More details can be found here: https://statuspage.incident.io/clickhousecloud/incidents/01K2FF662XXR72MXSHGZ1DG1T2#updates We continue to monitor the situation closely and will provide further updates as they become available.

Following the resolution of the database service incident, inventory screens and bulk remediation functionality have been fully restored. We are currently processing the asset updates that were delayed during the incident. This issue will be closed shortly.

Asset updates have been successfully completed, with no data loss observed.

We noticed a degraded performance around: identities posture page domains inventory our engineers are investigating and will update.

Currently, no new events are displayed on the Events page. Workflows are not affected.

Delay on events page, NOT affecting WF

We are currently investigating this issue.

Hey, Users might have issues connecting to the docontrol dashboard. We're investigating and will update soon.

We are currently experiencing inconsistencies with the delivery of events from the Google Admin Events API. Events may be delayed until the issue is resolved on Google's side. We have already escalated this matter to Google's API engineering team and are awaiting further updates from them. Thank you for your patience.

Issue resolved on Google's side.

We are experiencing issue of not getting part of the events, google is investigating with us why is it happening.

Google's engineering team is investigating this issue with a high-priority ticket.

Google acknowledged they got similar reports from different customers, working on investigating and solving it