- Detected by Pingoru
- Sep 15, 2026, 10:07 AM UTC
- Resolved
- Sep 16, 2026, 09:07 PM UTC
- Duration
- 1d 10h
Affected: Grafana Cloud
Timeline · 5 updates
-
investigating Sep 15, 2026, 10:07 AM UTC
We are currently seeing log ingestion issues, where some logs are dropped. We are investigating.
-
identified Sep 15, 2026, 11:57 AM UTC
We have identified the issue. We haven't fixed the cause yet, but things are certainly better now.
-
monitoring Sep 15, 2026, 07:32 PM UTC
Backfilling of the logs that could be rescued has been completed. Looking into the possibility of discovering what pods have missing logs from the time period 2 PM CEST till 15:00 PM CEST ish. Capabilities that hasn't had all of their logs backfilled will be notified if possible.
-
monitoring Sep 16, 2026, 12:43 PM UTC
Affected Capabilities have been notified via email. There's still 3 nodes that are occasionally struggling, a fix for those are being rolled out.
-
resolved Sep 16, 2026, 09:07 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Aug 20, 2026, 10:42 AM UTC
- Resolved
- Aug 20, 2026, 01:33 PM UTC
- Duration
- 2h 51m
Timeline · 3 updates
-
investigating Aug 20, 2026, 10:42 AM UTC
We are currently investigating this issue.
-
monitoring Aug 20, 2026, 11:11 AM UTC
The node has been marked unhealthy. A new node was spun up and took over the workloads.
-
resolved Aug 20, 2026, 01:33 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Jun 12, 2026, 12:21 PM UTC
- Resolved
- Jul 15, 2026, 11:28 AM UTC
- Duration
- 32d 23h
Affected: Capability provisioningAAD-AWS-Sync
Timeline · 2 updates
-
identified Jun 12, 2026, 12:21 PM UTC
Due to removal of permissions for Exchange Online, we are currently not able to create distribution groups and email aliases. The effects of this is that Capabilities created after that point are unable to receive emails from AWS and other third-party providers. We're awaiting details from the team managing Exchange Online as to why this is now broken..
-
resolved Jul 15, 2026, 11:28 AM UTC
Resolved for now.
Read the full incident report →
- Detected by Pingoru
- May 18, 2026, 09:32 AM UTC
- Resolved
- May 18, 2026, 09:38 AM UTC
- Duration
- 6m
Affected: Capability self-service portal (Blaster)
Timeline · 3 updates
-
investigating May 18, 2026, 09:32 AM UTC
Selfservice portal is fully down. We are investigating.
-
investigating May 18, 2026, 09:32 AM UTC
We are continuing to investigate this issue.
-
resolved May 18, 2026, 09:38 AM UTC
Problem resolved
Read the full incident report →
- Detected by Pingoru
- Feb 13, 2026, 12:35 PM UTC
- Resolved
- Feb 13, 2026, 01:04 PM UTC
- Duration
- 28m
Affected: Kubernetes [Hellman] - Capacity/Scheduling
Timeline · 4 updates
-
investigating Feb 13, 2026, 12:35 PM UTC
We are currently investigating this issue.
-
identified Feb 13, 2026, 12:37 PM UTC
The node is cordoned off and now marked as unhealthy. Once a new node has been started up, we will start draining all the workloads on this node. The root cause is the kubelet having problems with managing Pod lifecycle, in particular termination, so if draining adds to the problem, all workloads on that node will be forcefully terminated. Downtime can be expected.
-
monitoring Feb 13, 2026, 12:46 PM UTC
All workloads on the faulty node have now started up on a new node.
-
resolved Feb 13, 2026, 01:04 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Dec 01, 2025, 10:15 AM UTC
- Resolved
- Dec 01, 2025, 01:40 PM UTC
- Duration
- 3h 25m
Timeline · 1 update
-
resolved Dec 01, 2025, 01:50 PM UTC
In an attempt to optimise costs for metrics we accidentally ended up not gathering metrics for quite a significant amount of services for a few hours (11:15 CET till 14:40 CET) due to a bug that was triggered by our change. Apologies.
Read the full incident report →
- Detected by Pingoru
- Nov 28, 2025, 09:52 AM UTC
- Resolved
- Nov 28, 2025, 01:27 PM UTC
- Duration
- 3h 35m
Affected: Capability self-service portal (Blaster)Capability provisioningTopic management
Timeline · 3 updates
-
investigating Nov 28, 2025, 09:52 AM UTC
Several offerings on the Selfservice Platform are currently unavailable. This includes but is not necessarily limited to - approving memberships - creating new capabilities successfully - requesting Azure Resource groups We are working to resolve this as quickly as possible.
-
investigating Nov 28, 2025, 01:24 PM UTC
We are continuing to investigate this issue.
-
resolved Nov 28, 2025, 01:27 PM UTC
A problem with our Kafka setup prevented the consumption of events leading to the deteriorated state. This has now been addressed
Read the full incident report →
Notice September 30, 2025 - Detected by Pingoru
- Sep 30, 2025, 09:40 AM UTC
- Resolved
- Sep 30, 2025, 12:20 PM UTC
- Duration
- 2h 39m
Affected: Kubernetes [Hellman] - Capacity/SchedulingKubernetes [Hellman] - Ingress/Networking
Timeline · 4 updates
-
investigating Sep 30, 2025, 09:40 AM UTC
We are troubleshooting the issues ASAP.
-
identified Sep 30, 2025, 10:01 AM UTC
We're in the process of restoring the nodes that were faulty.
-
monitoring Sep 30, 2025, 10:14 AM UTC
New nodes are using correct launch template now, monitoring the situation.
-
resolved Sep 30, 2025, 12:20 PM UTC
The issues should be now resolved.
Read the full incident report →
- Detected by Pingoru
- Sep 08, 2025, 07:32 AM UTC
- Resolved
- Sep 08, 2025, 09:13 AM UTC
- Duration
- 1h 40m
Affected: AWS ec2-eu-central-1AWS elb-eu-central-1AWS s3-eu-central-1
Timeline · 3 updates
-
investigating Sep 08, 2025, 07:32 AM UTC
We are currently investigating this issue.
-
monitoring Sep 08, 2025, 07:41 AM UTC
A fix has been implemented and we are monitoring the results.
-
resolved Sep 08, 2025, 09:13 AM UTC
Fixed by AWS - https://health.console.aws.amazon.com/health/home#/account/event-log?End%20time=2025-09-07T11%3A12%3A52%20to%202025-09-08T11%3A12%3A52&Event%20category=issue&Status=closed&eventID=arn:aws:health:global::event/MANAGEMENTCONSOLE/AWS_MANAGEMENTCONSOLE_OPERATIONAL_ISSUE/AWS_MANAGEMENTCONSOLE_OPERATIONAL_ISSUE_4EEBD_2A76BAF307B&eventTab=details
Read the full incident report →
- Detected by Pingoru
- Aug 27, 2025, 06:03 AM UTC
- Resolved
- Aug 27, 2025, 07:01 AM UTC
- Duration
- 58m
Affected: Capability self-service portal (Blaster)Capability provisioningTopic managementAAD-AWS-Sync
Timeline · 2 updates
-
identified Aug 27, 2025, 06:03 AM UTC
The issue has been identified and a fix is being implemented.
-
resolved Aug 27, 2025, 07:01 AM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- May 29, 2025, 10:38 AM UTC
- Resolved
- May 29, 2025, 12:35 PM UTC
- Duration
- 1h 56m
Timeline · 3 updates
-
identified May 29, 2025, 10:38 AM UTC
Due to metrics collector going OOM sporadically since yesterday, metrics shipped from Hellman has been partly missing in Grafana Cloud. Working on implementing a fix now.
-
monitoring May 29, 2025, 11:32 AM UTC
Fix implemented. Seems stable for now, but without having resolved the root cause I suspect it'll probably happen again. Will keep monitoring it.
-
resolved May 29, 2025, 12:35 PM UTC
Incident itself is resolved, but the actual origin of the issue not quite yet. If you experience further issues with no data don’t hesitate to reach out directly via Slack to emcla.
Read the full incident report →
- Detected by Pingoru
- May 21, 2025, 12:02 PM UTC
- Resolved
- May 21, 2025, 03:11 PM UTC
- Duration
- 3h 8m
Affected: Grafana Cloud
Timeline · 3 updates
-
investigating May 21, 2025, 12:02 PM UTC
Currently look into an issue where queries to the data source "grafanacloud-all-metrics" gives the error "authentication error: invalid authentication credentials"
-
monitoring May 21, 2025, 12:23 PM UTC
A fix has been implemented and we are monitoring the results.
-
resolved May 21, 2025, 03:11 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- May 19, 2025, 12:28 PM UTC
- Resolved
- May 19, 2025, 12:30 PM UTC
- Duration
- 1m
Timeline · 1 update
-
resolved May 19, 2025, 12:28 PM UTC
A new feature was enabled for build.dfds.cloud mid Friday. Unfortunately something was missing in the production deployment and it caused certain events to not be triggered. In total 6 events has been pending from that time until today 2:15 PM CEST. They are no longer pending, so previously created resources like Kafka topics or Capability membership changes should now have been applied. Apologies for the interruption.
Read the full incident report →
- Detected by Pingoru
- Apr 14, 2025, 01:12 PM UTC
- Resolved
- Apr 15, 2025, 01:17 PM UTC
- Duration
- 1d
Affected: Capability self-service portal (Blaster)Capability provisioning
Timeline · 4 updates
-
investigating Apr 14, 2025, 01:12 PM UTC
We are currently investigating this issue.
-
identified Apr 14, 2025, 01:14 PM UTC
The issue has been identified and a fix is being implemented.
-
monitoring Apr 14, 2025, 02:49 PM UTC
Hotfixed applied, monitoring.
-
resolved Apr 15, 2025, 01:17 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Apr 08, 2025, 07:22 AM UTC
- Resolved
- Apr 10, 2025, 07:06 AM UTC
- Duration
- 1d 23h
Affected: Kafka metrics exporter
Timeline · 3 updates
-
investigating Apr 08, 2025, 07:22 AM UTC
Same as the incident a few weeks ago: https://dfdsit.statuspage.io/incidents/vllndmlym8xd Looking into it. Hopefully a more permanent solution this time. If you have alerts that are currently triggering because of the lack of data from this exporter, a temporary measure could be to set "Alert state if no data or all values are null" to "Keep Last State" for those alerts. It's not ideal, but it's much better than muting/silencing the alert itself. See screenshot: https://blob.juna.seq.tf/pub/vrsWnTTVXk37Srs.jpg
-
monitoring Apr 09, 2025, 04:04 PM UTC
A fix has been implemented and we are monitoring the results.
-
resolved Apr 10, 2025, 07:06 AM UTC
Since fix was implemented not a single scrape has been missed. Will continue to monitor for the next few days If it becomes problematic again, a different exporter is currently being evaluated to replace the current one.
Read the full incident report →
- Detected by Pingoru
- Mar 17, 2025, 06:30 PM UTC
- Resolved
- Mar 17, 2025, 06:30 PM UTC
- Duration
- –
Timeline · 1 update
-
resolved Mar 18, 2025, 08:44 AM UTC
A configuration change to Traefik which routes http traffic into the Kubernetes cluster caused endpoints that have /api in the path to return 404 not found to requests. The incident was resolved at ~00:30
Read the full incident report →
- Detected by Pingoru
- Mar 14, 2025, 07:53 AM UTC
- Resolved
- Mar 14, 2025, 10:40 AM UTC
- Duration
- 2h 47m
Affected: Kafka metrics exporter
Timeline · 3 updates
-
investigating Mar 14, 2025, 07:53 AM UTC
Currently looking into metrics from kafka-metrics having no data frequently. Currently the most probable cause is a big increase in consumer groups per topic for a few topics leading to much longer scrape times, resulting in a timeout.
-
monitoring Mar 14, 2025, 09:49 AM UTC
Issue seem potentially related to slower response times from Confluent Cloud. This seems to be improving now and the exporter can seemingly reliably export metrics within the set scrape threshold. Will continue to monitor the situation.
-
resolved Mar 14, 2025, 10:40 AM UTC
Seems stable... for now.
Read the full incident report →
- Detected by Pingoru
- Mar 03, 2025, 04:30 PM UTC
- Resolved
- Mar 03, 2025, 04:30 PM UTC
- Duration
- –
Timeline · 2 updates
-
resolved Mar 03, 2025, 08:53 PM UTC
Some Kubernetes pods failed to schedule due to available Node CPU and Memory capacity
-
postmortem Mar 18, 2025, 08:55 AM UTC
[https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)
Read the full incident report →
- Detected by Pingoru
- Mar 03, 2025, 01:32 PM UTC
- Resolved
- Mar 06, 2025, 08:37 AM UTC
- Duration
- 2d 19h
Affected: Kubernetes [Hellman] - Capacity/Scheduling
Timeline · 4 updates
-
investigating Mar 03, 2025, 01:32 PM UTC
We are currently investigating an issue that is causing the odd Kubernetes worker node to become unresponsive and evict its workloads to other nodes
-
monitoring Mar 03, 2025, 06:49 PM UTC
We have implemented a potential fix and are monitirong the results
-
resolved Mar 06, 2025, 08:37 AM UTC
Nodes have been stable since the fix was applied so we consider the issue resolved
-
postmortem Mar 18, 2025, 08:54 AM UTC
[https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)
Read the full incident report →
Critical January 13, 2025 - Detected by Pingoru
- Jan 13, 2025, 02:47 PM UTC
- Resolved
- Jan 13, 2025, 03:03 PM UTC
- Duration
- 15m
Affected: Capability self-service portal (Blaster)Capability provisioningTopic management
Timeline · 3 updates
-
investigating Jan 13, 2025, 02:47 PM UTC
We are investigating why the selfservice API is crash looping and causing the selfservice portal to not load properly.
-
monitoring Jan 13, 2025, 02:51 PM UTC
An invalid kafka message has been removed.
-
resolved Jan 13, 2025, 03:03 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Dec 04, 2024, 10:09 AM UTC
- Resolved
- Dec 04, 2024, 01:10 PM UTC
- Duration
- 3h 1m
Affected: Grafana Cloud
Timeline · 4 updates
-
investigating Dec 04, 2024, 10:09 AM UTC
Data and CET stacks are confirmed down. We will contact Grafana Support.
-
identified Dec 04, 2024, 10:14 AM UTC
The issue is confirmed by Grafana Support. Follow https://status.grafana.com/incidents/dr2xbss3cxns for updates.
-
monitoring Dec 04, 2024, 11:49 AM UTC
A fix has been implemented and we are monitoring the results.
-
resolved Dec 04, 2024, 01:10 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Nov 14, 2024, 02:19 PM UTC
- Resolved
- Nov 14, 2024, 02:37 PM UTC
- Duration
- 18m
Affected: Kubernetes [Hellman] - Developer accessAAD-AWS-Sync
Timeline · 2 updates
-
investigating Nov 14, 2024, 02:19 PM UTC
Currently looking into an issue where some Capabilities can't be accessed in Kubernetes via SSO. Pipeline access should still be operational.
-
resolved Nov 14, 2024, 02:37 PM UTC
False positive.
Read the full incident report →
