Northflank Outage History

This incident has been resolved.

Type: Incident Duration: 2 hours and 2 minutes Affected Components: Addons, Services May 10, 18:30:00 GMT+0 - Identified - We are currently experiencing a large DDoS attack in the Europe - West - Netherlands region. This is affecting public ingress into the cluster. Internal traffic is not affected. The team is working on additional mitigations. May 10, 20:10:07 GMT+0 - Monitoring - We have put in place mitigations and are monitoring the affected region. May 10, 20:31:56 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 36 minutes Affected Components: Certificates May 8, 19:00:00 GMT+0 - Identified - [Let's Encrypt is currently experiencing a partial service disruption](https://letsencrypt.status.io/) leading to certificate generation delays and failures on the Northflank platform. This will impact provisioning of new addons, domains and BYOC clusters. May 8, 21:36:09 GMT+0 - Resolved - Certificate generation is functioning normally again. We will continue to monitor the situation.

Type: Incident Duration: 52 minutes Affected Components: Addons, Services Apr 8, 06:15:00 GMT+0 - Identified - We are aware of an issue with a node affecting some stateful workloads in US - Central. We are working on a fix for this incident. Apr 8, 07:07:27 GMT+0 - Resolved - We have recovered the node, and all workloads are now running again.

Type: Incident Duration: 34 minutes Affected Components: Northflank App Mar 16, 18:16:28 GMT+0 - Resolved - This incident has been resolved. Mar 16, 18:15:58 GMT+0 - Monitoring - We identified the root cause as a system component causing disproportionate database load. This is affecting read performance leading to degraded performance for the app UI. We have implemented a fix and are monitoring the situation. Mar 16, 17:42:54 GMT+0 - Identified - We are investing an incident where the UI is failing to load correctly.

Type: Incident Duration: 29 minutes Affected Components: Networking, Addons, Services Mar 9, 12:06:32 GMT+0 - Investigating - We are currently investigating this incident. Mar 9, 12:35:56 GMT+0 - Resolved - The root cause has been identified as an issue with the GCP infrastructure control plane and has been resolved. There was minimal impact to running user workloads. The main impact was a delay in provisioning net new or redeploying existing workloads.

Type: Incident Duration: 1 hour and 34 minutes Affected Components: Builds Feb 19, 15:00:00 GMT+0 - Identified - We are aware of an issue where builds fail to start when using the Heroku 24 buildpack image. We are currently working on identifying a solution. We will provide an update when the fix has been released. Feb 19, 16:34:09 GMT+0 - Resolved - We have released a fix for the issue.

Type: Incident Duration: 39 minutes Affected Components: Networking, , Addons, Jobs, Services, Northflank Platform → Dec 11, 13:54:32 GMT+0 - Identified - There are currently issues with node stability in the London region leading to partial outages for some workloads. The team has identified the issue and is implementing a mitigation. Dec 11, 14:13:28 GMT+0 - Monitoring - The mitigation is in place and the region is operating as expected. Dec 11, 14:33:03 GMT+0 - Resolved - The incident has been resolved and the cause identified. A node image security release lead to the host filesystem going into a read only mode when interacting with a specific set of workloads causing the node to become unresponsive. We have implemented a mitigation and are working on a permanent solution.

Type: Incident Duration: 1 hour and 44 minutes Affected Components: Addons, Jobs, Services Sep 24, 23:53:50 GMT+0 - Identified - Dockerhub is currently experiencing issues related to authentication. This may cause issues with starting jobs, addons and services. Sep 25, 01:11:08 GMT+0 - Monitoring - We are observing that image pull requests for Docker Hub images are now succeeding. Sep 25, 01:38:09 GMT+0 - Resolved - Dockerhub is fully operational and image pulling is now working as expected.

Type: Incident Duration: 18 hours and 29 minutes Affected Components: Addons, Services, Certificates Sep 16, 15:09:51 GMT+0 - Monitoring - Our DNS provider NS1 is experiencing delays in DNS propagation. This is affecting service and addon creation. NS1 status page: Sep 17, 09:39:01 GMT+0 - Resolved - The incident has been resolved on NS1 and we have not been seeing any more related failures over the last hour.

Type: Incident Duration: 1 hour and 17 minutes Affected Components: Builds Aug 25, 21:48:19 GMT+0 - Identified - There is an issue with the underlying storage which provides the local disk-based caching feature for builds on Northflank's infrastructure. BYOC builds and builds without a local cache will be unaffected. We are currently working on a fix for this. Aug 25, 22:45:24 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 25, 23:04:54 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour Affected Components: Logs and Metrics Aug 5, 07:23:59 GMT+0 - Investigating - We are currently investigating this incident. Aug 5, 07:55:18 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 5, 08:23:56 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 17 minutes Affected Components: Logs and Metrics Jun 11, 22:00:00 GMT+0 - Identified - Metrics query and ingestion are currently affected. Logs are unaffected. We are currently applying a fix for the issue and monitoring the results. Jun 11, 23:16:34 GMT+0 - Monitoring - The fix has been rolled out and we are currently monitoring the ingestion rate and catch-up. Jun 12, 00:16:31 GMT+0 - Resolved - The ingestion rate has been stable for 30 minutes and historic writes have caught up. We sincerely apologise for the service disruption you may have experienced. We are investigating the root cause to prevent similar issues in the future.

Type: Incident Duration: 1 hour and 17 minutes Affected Components: Addons Nov 8, 19:20:22 GMT+0 - Resolved - This incident has been resolved. Nov 8, 18:02:57 GMT+0 - Monitoring - Google Cloud is experiencing issues with snapshot upload in the US Central region. This is causing delays for addon snapshot backups within the region

Type: Incident Duration: 1 hour and 27 minutes Affected Components: Services, Addons, Certificates Oct 11, 18:41:37 GMT+0 - Monitoring - Our DNS provider is experiencing issues with DNS propagation leading to delays in provisioning new workloads and certificates. Existing workloads are unaffected. Oct 11, 20:09:04 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Affected Components: Logs and Metrics Jul 16, 10:46:00 GMT+0 - Resolved - Metrics experienced ingestion issues between 10:46AM -> 11:30AM UTC We implemented a fix and are currently monitoring the result.

Type: Incident Affected Components: Logs and Metrics Jul 16, 04:00:00 GMT+0 - Resolved - Engineers identified an incident with Northflank's logging backend between 0400UTC and 0700UTC. The problem has been resolved. The impact is log loss between this timeframe. We are still investigating the cause of the issue and will keep you updated and whether logs can be recovered.