Whalebone Outage History

We are investigating an issue causing DNS Traffic dashboard data not to load correctly in the Admin Portal. Some customers may see missing or unavailable DNS Traffic dashboard data. DNS resolution, filtering, and protection remain operational; this issue affects portal visibility only. Our team is investigating and will provide another update when more information is available.

We are investigating delayed DNS traffic data in EU-01. Some customers may see slow-loading or missing DNS traffic data in the Admin Portal. DNS Traffic related alerting is also affected. Our team has identified a backlog in the DNS traffic data pipeline and is working on recovery. Based on current information, the impact is limited to DNS traffic visibility and related alerting.

We are continuing to recover delayed DNS traffic data in EU-01. The underlying storage cluster is healthy, and delayed traffic records are being backfilled. Some customers may still see delayed or missing DNS traffic data in the Admin Portal. DNS Traffic related alerting may also remain affected until the backlog is processed. We currently expect this to take approximately 12 hours. No customer action is required.

We are investigating a data visibility issue affecting Resolver metrics, Resolver alerts, and the Threats dashboard. Customers may currently be unable to view Resolver metrics, Resolver alerts may be delayed, and the Threats dashboard may show no data. DNS resolution and filtering remain operational. Our team is working to restore the affected data backend. We will provide another update as soon as more information is available.

We are seeing recovery of the affected data backend. Resolver metrics and Threats dashboard data are starting to become available again. Some Resolver metrics or alerts may still be delayed while data processing catches up. DNS resolution and filtering remain operational. Our team is continuing to monitor the recovery and will provide another update once service is fully restored.

We are investigating a false positive affecting Content Filtering. Some customers may be unable to access Microsoft sign-in through login.microsoftonline.com when the Gambling category is active in Content Filtering. General DNS resolution remains operational. Protection and filtering outside this false positive are not known to be affected. Our team is investigating and working to mitigate the incorrect blocking decision. Next update: within 60 minutes.

We have identified the source of the incorrect Content Filtering categorization for login.microsoftonline.com and have pushed corrective updates. The domain should no longer be blocked on resolvers that have accepted the latest update. We are forcing regeneration of the remaining filtering data and verifying propagation. General DNS resolution remains operational. Protection and filtering outside this false positive are not known to be affected. Next update: within 60 minutes.

We are investigating an issue affecting the Resolvers metrics dashboard in the Admin Portal. Some customers may currently see missing metrics for the last few hours. The issue appears to be related to delayed or missing data in the resolver metrics pipeline. DNS resolution and protection remain operational; this issue affects dashboard visibility only. We will share another update as soon as more information is available.

We have identified the issue affecting the Resolvers metrics dashboard in the Admin Portal and applied mitigation. Resolver metrics are now flowing again and missing data is being backfilled. DNS resolution and protection were not affected during this incident; the impact was limited to dashboard visibility. We are monitoring the situation to confirm full recovery before resolving the incident.

We are currently investigating service degradation affecting the Management API. Our team is actively working to identify the cause and restore full service as quickly as possible. Based on our current investigation, this issue is connected to degraded performance reported by our infrastructure provider. https://exoscalestatus.com/incidents/230257 We will share the next update as soon as we have more confirmed information.

The Management API is recovering and service is gradually returning. We are continuing to monitor the situation closely to ensure stability before closing the incident. Some customers may still experience intermittent issues during recovery. For more details on recovery you can follow our infrastructure provider updates https://exoscalestatus.com/incidents/230257

Customers in the Main region may notice intermittent gaps or missing data in the DNS Traffic (PassiveDNS) section of the Whalebone Admin Portal or timeouts when calling API. Affected Services: DNS Traffic dashboards and logs in the portal, API timeouts Unaffected Services: DNS resolution remains fully operational and is not impacted. Other regions are not impacted by this specific issue. We have implemented several infrastructure adjustments to mitigate the impact, and our team is currently working on a permanent fix while continuing to investigate. Next Update: We will provide further information once the permanent fix is deployed and verified. We apologize for any inconvenience this may cause to your services.

We are investigating an issue affecting the DNS4EU public resolver in Milan where some .org domain queries may have returned incorrect DNS responses. As a mitigation, we have removed the affected node from service. We are analyzing logs to determine scope and root cause. Next update in 30 minutes (or sooner).

Mitigation remains in place (Milan node out of rotation). Service operating normally via remaining resolvers. Investigation into duration/scope/root cause continues. Next update in 60 minutes (or sooner).

We are preparing to return the Milan DNS4EU resolver node back into service. The node has been checked in isolation and the incorrect .org resolution observed earlier is no longer occurring — DNS queries are resolving correctly now. As a precaution, we have added additional monitoring and alerts so we can quickly detect and respond if the issue happens again. Our investigation is ongoing. Next update will be shared once we have identified the root cause or have more information to share.

We have identified an issue where selected regulatory domains—primarily in the Czechia and Poland regions—that should be blocked based on regulatory feeds are currently not being blocked. We are actively investigating the affected domains and whether other regulatory lists or regions may also be impacted. Impact Regulatory domains that should be blocked may still resolve normally for some customers. Other DNS resolution and protection services remain operational. What We’re Doing Our engineering and operations teams are actively investigating the underlying cause of this enforcement gap. We are reviewing the regulatory list propagation and enforcement logic across resolver regions. Next Update We expect to provide the next update as soon as new information becomes available.

We are investigating an issue causing intermittent Management API timeouts and, in some cases, responses like “invalid combination of access key and secret key.” Impact appears limited to a subset of customers. Our upstream provider is reporting increased network latency/packet loss in the same region (their status page), which is causing these timeouts. No action is required at this time — please do not rotate keys unless instructed by support (the message can be misleading during this degradation). Next update: in 30 minutes (or sooner if status changes).

We’re investigating an issue in EU-01 where the Admin Portal → Resolvers list may load very slowly or fail to load.

We are investigating an issue causing delayed or missing dashboard data for some customers. This impacts Threat data visibility and DNS traffic dashboards (data may be delayed by ~1 hour or more).