Cognite Outage History

Started Jun 04, 2026, 06:34 PM UTC· resolved Jun 18, 2026, 10:19 AM UTC

1. investigating Jun 04, 2026, 06:34 PM UTC

   We are currently investigating an issue where Data Modeling search and aggregate results are not refreshing as expected on the az-eastus-1 cluster. Users may experience incorrect or zero counts in search results, and newly ingested data may not appear in the CDF Search UI. Our engineering team has identified the root cause and a fix is currently in progress. We will provide further updates as the situation develops.

2. identified Jun 04, 2026, 07:10 PM UTC

   We have identified the cause of an issue affecting Data Modeling search and aggregate results on the az-eastus-1 cluster. This issue is limited to a specific data pattern and does not affect all customers only projects ingesting JSON data where one key is used as a prefix before a dot (.) in another key within the same JSON object are impacted. Workaround: To avoid triggering this issue, please ensure JSON objects do not contain keys where one key shares a prefix with another key before a dot (.) or end of string. For example, a structure where one key appears as a prefix of another dot-separated key should be avoided. Our engineering team is actively working on a permanent fix. We will provide further updates as the situation develops.

3. identified Jun 05, 2026, 07:29 AM UTC

   A bug in the Elasticsearch mapping refresh is causing search and aggregate results to not update for projects using JSON properties where a plain key and a dotted key share a prefix (e.g. "min" and "min.A" in the same JSON object). A first fix was insufficient; a second fix (PR #1908) is now in pipeline and rollout to az-eastus-1 is expected within ~1 week, pending testing. Example of triggering data pattern: { "min": "value", "min.A": "nested_value" } Any JSON property where a plain key (min) and a dotted key (min.A) share the same prefix will trigger the naming collision. Currently fixing is in progress.

4. identified Jun 05, 2026, 01:02 PM UTC

   The fix for the Elasticsearch mapping bug has been deployed to staging and initial results look good. Pending successful staging validation, rollout to production is planned throughout next week. The workaround remains to avoid using JSON keys with shared prefixes and dotted variants.

5. monitoring Jun 11, 2026, 11:55 AM UTC

   The fix has been applied and indexing has resumed. We are currently monitoring the state of the fix at this stage.

6. resolved Jun 18, 2026, 10:19 AM UTC

   The issue has now been fully contained and marked as resolved.

Read the full incident report →

Started Jun 04, 2026, 01:30 PM UTC· resolved Jun 04, 2026, 02:20 PM UTC

1. investigating Jun 04, 2026, 01:30 PM UTC

   Data Workflow is unavailable in cluster az-tyo-gp-001, impacting workflow execution and related operations. Investigation and recovery efforts are ongoing.

2. resolved Jun 04, 2026, 02:20 PM UTC

   Data Workflow availability has been fully restored in cluster az-tyo-gp-001. Validation and monitoring confirm normal operation, and the incident has been resolved and closed.

Read the full incident report →

Started Jun 03, 2026, 01:08 PM UTC· resolved Jun 03, 2026, 02:52 PM UTC

1. investigating Jun 03, 2026, 01:08 PM UTC

   We have identified a potential issue surrounding AI Chat agent, where the agent selector dropdown is rendering with heavily overlapped/duplicated text, making it effectively unusable. The issue is a frontend rendering regression, internally detected at 14:35 CEST today.

2. identified Jun 03, 2026, 01:13 PM UTC

   We have identified a potential issue surrounding AI Chat agent, where the agent selector dropdown is rendering with heavily overlapped/duplicated text, making it effectively unusable. The issue is a frontend rendering regression, internally detected at 14:35 CEST today.

3. resolved Jun 03, 2026, 02:52 PM UTC

   The incident is now resolved as the fix was deployed to mitigate the UI rendering issue in the agent chat.

Read the full incident report →

Started Jun 01, 2026, 07:35 AM UTC· resolved Jun 01, 2026, 08:08 AM UTC

1. investigating Jun 01, 2026, 07:35 AM UTC

   We have identified that Polyfill.io is asking for credentials on page load for hub.cognite.com the team is still investigating this issue.

2. resolved Jun 01, 2026, 08:08 AM UTC

   The team has identified the root cause of the issue and successfully deployed a fix. As a result, the popup is no longer appearing, and normal functionality has been restored. This incident has been fully resolved.

Read the full incident report →

Started May 28, 2026, 09:42 AM UTC· resolved Jun 08, 2026, 09:52 AM UTC

1. investigating May 28, 2026, 09:42 AM UTC

   We are currently investigating reports of very slow queries when fetching instances. Our engineering team is actively analyzing the issue and working to identify the root cause.

2. investigating May 28, 2026, 10:48 AM UTC

   The issue has been mitigated and service performance has returned to normal. Our team has confirmed the issue is resolved and will continue to monitor the environment to ensure stability.

3. monitoring May 28, 2026, 03:38 PM UTC

   The issue has been mitigated and service performance has returned to normal. Our team has confirmed the issue is resolved and will continue to monitor the environment to ensure stability.

4. resolved Jun 08, 2026, 09:52 AM UTC

   The final rollout of the fix for intermittent Data Modeling performance degradation on az-eastus-1 completed successfully, with no significant latency spikes observed during deployment. Application performance is stable and we have not observed any further issues. This incident is now considered resolved.

Read the full incident report →

Started May 27, 2026, 04:03 PM UTC· resolved May 27, 2026, 09:10 PM UTC

1. investigating May 27, 2026, 04:03 PM UTC

   The asset tree scroll function in Cognite InField's Create Observation flow is frozen, preventing users from scrolling the asset list or selecting assets via checkbox, which blocks observation creation. Currently, the issue is being investigated.

2. identified May 27, 2026, 06:13 PM UTC

   The cause of the incident has been identified and the fix is currently in development. We expect the fix to be deployed as soon as possible.

3. monitoring May 27, 2026, 09:07 PM UTC

   The team has successfully deployed a fix for the issue. The functionality is back to normal. We are moving to the monitoring stage.

4. resolved May 27, 2026, 09:10 PM UTC

   The fix was successful in mitigating the issue. Furthermore, no issues were observed during the Monitoring period.

Read the full incident report →

Started May 21, 2026, 07:10 PM UTC· resolved May 22, 2026, 05:56 PM UTC

1. investigating May 21, 2026, 07:10 PM UTC

   We are currently investigating periodic instability affecting Data Modeling Storage (DMS) services on the az-eastus-1 cluster. Users may experience intermittent 5XX errors when accessing Data Modeling endpoints. Our engineering team is actively investigating.

2. investigating May 21, 2026, 07:55 PM UTC

   Problem: The datamodelstorage service in az-eastus-1 is experiencing periodic spikes to 100% 5xx error rates, resulting in major outages for users. Impact: Multiple customers are receiving repeated 503 errors. All users on this cluster are likely affected. Sev A declared.

3. identified May 21, 2026, 08:20 PM UTC

   The engineers have identified the root cause and are working on a solution.

4. identified May 21, 2026, 08:26 PM UTC

   Engineers have identified a mitigation action and are working on a temporary solution, as well as monitoring the situation.

5. monitoring May 21, 2026, 08:52 PM UTC

   Cognite has reviewed deployment history, logs, and network dependencies. An internal process was found to be blocking operations. Cognite has stopped that process. The rates on the 500 errors has dropped, and the responsible engineering teams will resume investigation tomorrow.

6. monitoring May 22, 2026, 02:37 PM UTC

   Cognite has deployed a fix for the issue causing intermittent 5xx errors on the Data Modeling Service (DMS) in the az-eastus-1 cluster. All related SLO alerts have cleared and background tasks have resumed normal operation. We are continuing to observe the cluster closely before resolving the incident. We will post a further update once monitoring confirms full stability.

7. resolved May 22, 2026, 05:56 PM UTC

   The issue affecting Data Modeling Storage (DMS) services on the az-eastus-1 cluster has been resolved. All services are now operating normally.

Read the full incident report →

Started May 21, 2026, 11:36 AM UTC· resolved May 22, 2026, 06:27 AM UTC

1. investigating May 21, 2026, 11:36 AM UTC

   The incident was caused by an issue preventing Fusion Data Modeling from loading in CDF. Engineers are currently investigating and working toward a resolution.

2. monitoring May 21, 2026, 12:14 PM UTC

   The engineers have confirmed that the issue has been resolved and are currently monitoring the service to ensure everything returns to normal operation.

3. resolved May 22, 2026, 06:27 AM UTC

   The issue is resolved now. During the monitoring period, there were no issues observed.

Read the full incident report →

Started May 12, 2026, 09:44 AM UTC· resolved May 12, 2026, 07:56 PM UTC

1. investigating May 12, 2026, 09:44 AM UTC

   We are investigating elevated error rates affecting data modelling APIs, likely caused by request processing delays related to large data model schemas.

2. identified May 12, 2026, 07:51 PM UTC

   Our teams have identified the cause of the elevated error rates affecting data modeling APIs. Currently working towards resolving the issue.

3. monitoring May 12, 2026, 07:54 PM UTC

   Our team has deployed a fix for the elevated error rates in the data modeling API issue. We are moving to the monitoring period.

4. resolved May 12, 2026, 07:56 PM UTC

   The fix deployed to address the issue was successful. The metrics are back to normal on the error rates in the data modeling API. We are resolving the incident now.

Read the full incident report →

Started May 11, 2026, 11:26 AM UTC· resolved May 13, 2026, 02:51 PM UTC

1. investigating May 11, 2026, 11:26 AM UTC

   We noticed that function deployments are failing in westeurope-1 cluster. Our engineers are currently investigating the issue.

2. monitoring May 12, 2026, 10:47 AM UTC

   Function deployments across affected clusters have stabilized. No new deployment errors have been observed since May 11. We are actively monitoring deployments, which are now passing consistently. We will provide further updates once we receive additional information or observe any changes.

3. resolved May 13, 2026, 02:51 PM UTC

   Deployments have remained stable since May 11, and we have not observed any new deployment-related errors. Therefore, this incident has now been marked as resolved.

Read the full incident report →

Started May 06, 2026, 10:49 AM UTC· resolved May 06, 2026, 11:18 AM UTC

1. identified May 06, 2026, 10:49 AM UTC

   Creating or updating a Cognite Function on Azure is currently failing for functions whose requirements.txt "requirements.txt": http://requirements.txt/ lists cognite-sdk. Affected requests return "Function deployment failed unexpectedly."

2. monitoring May 06, 2026, 10:57 AM UTC

   The fix has been rolled out and the functions are being deployed. We will be keeping a tab on the deployment for time being.

3. resolved May 06, 2026, 11:18 AM UTC

   The fix has been deployed and all function deployment are completing.

Read the full incident report →

Started Apr 30, 2026, 10:38 AM UTC· resolved May 02, 2026, 06:04 AM UTC

1. investigating Apr 30, 2026, 10:38 AM UTC

   We are investigating an issue that may cause file upload statuses to appear delayed or incorrect. Our engineering team is actively working to identify the root cause and assess the overall impact.

2. identified Apr 30, 2026, 01:42 PM UTC

   The engineering team has identified the issue and is raising the relevant limits to reduce impact while closely monitoring progress.

3. identified Apr 30, 2026, 06:35 PM UTC

   Cognite has deployed a configuration change to double the DMS local concurrency limit per pod, aiming to reduce throttling and help clear the file syncing backlog in the az-eastus-1 cluster. The deployment is complete and we are monitoring relevant dashboards to assess its impact on file upload propagation and timeout rates.

4. monitoring May 01, 2026, 09:37 AM UTC

   The limit increases are effective, and we are observing stable conditions for the API. We will monitor the progress and will proceed further depending on the observations.

5. resolved May 02, 2026, 06:04 AM UTC

   The conditions are now stable for the API after observing the fixes applied, Therefore this incident is now resolved.

Read the full incident report →

Started Apr 29, 2026, 06:43 PM UTC· resolved Apr 30, 2026, 11:28 AM UTC

1. investigating Apr 29, 2026, 06:43 PM UTC

   Cognite is investigating a large slowdown on Raw storage in AZ-EASTUS-1. Slowdowns have been experienced by multiple customers. More information will follow as our investigation continues. ***UPDATE*** Cognite has deployed a new Raw DB to AZ-EASTUS-1. We are observing the load to the new database to see how much the new db will alleviate the slowness. This could take a bit of healing time. Cognite is also requesting updates from Microsoft on the Sev A case currently opened.

2. investigating Apr 29, 2026, 06:45 PM UTC

   ***UPDATE*** Cognite has deployed a new Raw DB to AZ-EASTUS-1. We are observing the load to the new database to see how much the new db will alleviate the slowness. This could take a bit of healing time. Cognite is also requesting updates from Microsoft on the Sev A case currently opened.

3. identified Apr 30, 2026, 03:31 AM UTC

   The immediate performance issue for new tables has been mitigated. Teams have confirmed a significant speed improvement when data is reingested into new tables, with successful runs of key workflows. However, existing tables remain on the original, slower cluster. Their performance is still degraded and is dependent on performance enhancements of a 3rd party service.

4. resolved Apr 30, 2026, 11:28 AM UTC

   The issue is resolved now and the service is back to normal.

Read the full incident report →

Started Apr 29, 2026, 01:19 PM UTC· resolved Apr 30, 2026, 11:01 AM UTC

1. investigating Apr 29, 2026, 01:19 PM UTC

   We have identified a potential issue when trying to open canvas you are prompted with insufficient access rights across projects in the cluster.

2. identified Apr 29, 2026, 02:59 PM UTC

   The team has identified the issue and a fix is currently on the way.

3. monitoring Apr 29, 2026, 03:20 PM UTC

   A fix has been rolled out now and we are currently monitoring the state of this incident.

4. resolved Apr 30, 2026, 11:01 AM UTC

   The issue has been resolved, and the service is now operating normally.

Read the full incident report →

Started Apr 27, 2026, 11:07 PM UTC· resolved Apr 27, 2026, 11:50 PM UTC

1. investigating Apr 27, 2026, 11:07 PM UTC

   Azure OpenAI models are currently experiencing degraded availability in certain regions. Our team is actively monitoring the situation and working to restore full service. Affected production clusters: • aw-was-gp-001 • az-eastus-1 • az-gru-001 • az-pnq-gp-001 • az-tyo-002 • az-tyo-gp-001

2. monitoring Apr 27, 2026, 11:30 PM UTC

   Azure OpenAI model availability has stabilized. We are no longer observing 500 errors from Azure, though our team continues to monitor closely.

3. resolved Apr 27, 2026, 11:50 PM UTC

   Azure OpenAI GPT model endpoints in the affected US regions have fully recovered. Multiple rounds of health checks confirmed all endpoints are responding successfully with no errors.

Read the full incident report →

Started Apr 24, 2026, 02:17 PM UTC· resolved Apr 25, 2026, 09:25 AM UTC

1. investigating Apr 24, 2026, 02:17 PM UTC

   Cognite is investigating k8s pod restarts in az-eastus-1. This is possibly affecting time series, searching, and contextualization capabilities. Updates will be provided when more information is gathered.

2. investigating Apr 24, 2026, 03:15 PM UTC

   We are seeing widespread DNS resolution failures in the az-eastus-1 cluster, affecting multiple pods and disrupting the timeseries service.

3. identified Apr 24, 2026, 03:40 PM UTC

   We have confirmed that all customers on az-eastus-1 are affected due to ongoing Azure platform issues in the East US region, now officially acknowledged by Microsoft. These issues impact provisioning, scaling, and connectivity for workloads, resulting in widespread DNS resolution failures and a large number of pods across multiple namespaces in CrashLoopBackOff. Customer impact includes service disruptions for timeseries and data modeling services, intermittent errors accessing InField, and degraded availability for multiple workloads and endpoints. Example error patterns include Postgres connection timeouts and DNS lookup failures.

4. monitoring Apr 24, 2026, 04:20 PM UTC

   Cognite has observed all services have recovered or are in a good state of recovery. Customers may possibly still see some slowness in apps recovering from the dns outage.

5. monitoring Apr 24, 2026, 11:26 PM UTC

   While our previous update indicated a good state of recovery, we have started to observe service interruptions again across the environment. Customers may continue to experience slowness or intermittent failures in applications as we navigate the final stages of stabilization following the DNS outage. Our engineering team is closely monitoring these new developments and working to ensure a consistent recovery. We will provide further updates as the situation stabilizes.

6. monitoring Apr 25, 2026, 12:43 AM UTC

   CDF has recovered again after recurrence of the DNS issue. Microsoft has confirmed they have now completed the rollback of the suspected root cause and we will continue monitoring the issue for a few more hours to verify CDF has fully recovered.

7. resolved Apr 25, 2026, 09:25 AM UTC

   Microsoft has confirmed the underlying issue has been mitigated and CDF has been running without issues for several hours.

Read the full incident report →

Started Apr 23, 2026, 09:21 AM UTC· resolved Apr 23, 2026, 01:23 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Apr 20, 2026, 08:24 AM UTC· resolved Apr 20, 2026, 11:13 AM UTC

1. monitoring Apr 20, 2026, 08:24 AM UTC

   There has been an issue due to which Custom Apps menu item were not visible in the Fusion Menu. Engineering team has deployed a fix and working on a rca.

2. resolved Apr 20, 2026, 11:13 AM UTC

   The issue has been resolved and the Custom Apps menu is visible and functional.

Read the full incident report →

Started Apr 20, 2026, 08:24 AM UTC· resolved Apr 20, 2026, 11:13 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Apr 17, 2026, 04:29 PM UTC· resolved Apr 24, 2026, 08:10 AM UTC

1. investigating Apr 17, 2026, 04:29 PM UTC

   Customers are seeing 35-40% of function deployments failing. Possible correlation to an issue with the files api returning large amounts of 429 codes.

2. investigating Apr 17, 2026, 04:31 PM UTC

   Currently doing triage on the API issue with incident commander.

3. identified Apr 17, 2026, 07:15 PM UTC

   Cognite has discovered a certificate that has expired that is impacting az-eastus-1 customers. More info to come as we work to renew the cert.

4. monitoring Apr 17, 2026, 08:45 PM UTC

   Our engineering team oncall was unable to renew the certificate at the time as he was not at home or in the office. We are hopeful that the cert will be renewed as soon as possible.

5. investigating Apr 17, 2026, 09:16 PM UTC

   We've noticed there are frequent function deployments failing in az-eastus-1 and we are investigating the issue. Due to the technically of the situation we are awaiting more information from relevant parties, and have paused the incident as we don't see any clear backend issues.

6. identified Apr 20, 2026, 08:04 PM UTC

   Cognite continues to work on this issue affecting function deployments. Currently, there is a hypothesis around deploying functions to entra virtual networks. We will provide updates as soon as progress is made.

7. monitoring Apr 21, 2026, 04:41 PM UTC

   Cognite will be placing this incident in monitoring status. Investigation has shifted to an issue with Microsofts AZ-EASTUS-1. It appears a vnet was deployed by Microsoft last week and was not available to it's customers.

8. resolved Apr 24, 2026, 08:10 AM UTC

   After carefully monitoring the implemented fixes, our engineering team has confirmed that the incident has been resolved.

Read the full incident report →

Started Apr 17, 2026, 04:29 PM UTC· resolved Apr 24, 2026, 08:10 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Apr 17, 2026, 01:45 AM UTC· resolved Apr 22, 2026, 01:27 PM UTC

1. investigating Apr 17, 2026, 01:45 AM UTC

   We are currently experiencing intermittent degradation in integration services, related to ongoing database maintenance in Microsoft Azure. Our engineers are actively investigating the issue with Azure to identify the root cause and will provide updates as more information becomes available.

2. monitoring Apr 17, 2026, 11:28 AM UTC

   The database maintenance has completed , services and database connectivity is restored. We are still monitoring the issue and waiting for an RCA from the Azure team.

3. resolved Apr 22, 2026, 01:27 PM UTC

   Our engineering team has completed monitoring this issue. The database maintenance has completed, services and database connectivity is restored. Therefore, we are resolving this incident.

Read the full incident report →

Started Apr 01, 2026, 07:44 AM UTC· resolved Apr 23, 2026, 05:53 PM UTC

1. investigating Apr 01, 2026, 07:44 AM UTC

   "We are currently investigating an issue affecting the Data Modeling service. Some users may experience degraded performance and increased error rates, including 408 Request Timeout errors. Our engineering team is actively working to resolve the issue. We will provide further updates as the situation develops. "

2. investigating Apr 01, 2026, 07:54 AM UTC

   Please be informed that DMS services will be temporarily unavailable during the scheduled maintenance window. Maintenance Window: 16:00 – 18:00 CEST / 23:00 – 01:00 JST Cluster: az-tyo-gp-001

3. monitoring Apr 01, 2026, 02:48 PM UTC

   The scheduled maintenance has been successfully completed, and DMS service has returned to normal operation. The engineering team is continuing to monitor the service to ensure stability.

4. investigating Apr 02, 2026, 08:26 AM UTC

   We have observed that DMS services remain unstable. Consequently, the engineering team has decided to conduct additional emergency maintenance. Please be informed that DMS services will be temporarily unavailable during the maintenance window. Emergency Maintenance Window: 17:00 – 18:00 CEST / 00:00 – 01:00 JST Cluster: az-tyo-gp-001

5. identified Apr 02, 2026, 04:01 PM UTC

   Our teams have completed the emergency maintenance successfully, the engineering team is continuing to monitor the service to ensure stability.

6. monitoring Apr 02, 2026, 08:25 PM UTC

   Service success rates have recovered to 100%, and the affected dashboards are now reporting normal operation. The team is continuing to monitor the situation closely

7. resolved Apr 23, 2026, 05:53 PM UTC

   Since the success rates have returned to 100%, and all affected dashboards are reporting normal operation. The issue has been fully resolved.

Read the full incident report →

Started Mar 31, 2026, 08:23 AM UTC· resolved Apr 15, 2026, 12:34 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Mar 20, 2026, 03:31 PM UTC· resolved Mar 20, 2026, 05:22 PM UTC

1. investigating Mar 20, 2026, 03:31 PM UTC

   We are seeing a downtime in Timeseries and Sequence component of the Cognite API.

2. monitoring Mar 20, 2026, 03:57 PM UTC

   Service has been restored to the timeseries and sequence API. Investigation continues into the cause of the service interruption.

3. resolved Mar 20, 2026, 05:22 PM UTC

   The incident is now resolved as the issue is contained after a period of monitoring.

Read the full incident report →