Railway Outage History

Type: Incident Duration: 1 hour and 24 minutes Affected Components: US East (Metal / Virginia, USA) Mar 25, 20:23:24 GMT+0 - Investigating - We are currently investigating this incident. Mar 25, 21:17:27 GMT+0 - Identified - We have traced this back to a change rolled out with static IP egress gateways. We have begun the rollbacks and are already seeing recovery. Mar 25, 21:29:49 GMT+0 - Monitoring - The rollback has completed, we are seeing recovery, and are currently monitoring the result. Mar 25, 21:47:30 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 15 minutes Affected Components: , External / Third-Party Integrations → Mar 25, 18:05:54 GMT+0 - Investigating - Dockerhub Registry Outage, pulls from Dockerhub may fail. Mar 25, 19:21:23 GMT+0 - Resolved - Dockerhub reports recovery

Type: Incident Duration: 1 hour and 45 minutes Affected Components: , Edge Network (Legacy) → Mar 25, 16:09:16 GMT+0 - Investigating - Cert discrepancy between Fastly and Railway leading to SAN errors. The network team is actively triaging the issue for affected customers. Mar 25, 16:11:10 GMT+0 - Identified - For some affected domains, cert provisioning is inoperable, this is due to incorrect domain configuration, please re-add your domains to triage the issue. Mar 25, 16:38:28 GMT+0 - Identified - Manual workarounds for customers show that access is restored, we are working with our upstream vendor to implement domain forwarding to mitigate customer misconfiguration. Mar 25, 17:54:18 GMT+0 - Resolved - Network behavior is back to nominal.

Type: Incident Duration: 12 minutes Affected Components: Dashboard (railway.com) Mar 25, 15:55:01 GMT+0 - Investigating - Railway Dashboard Load Times Impacted Mar 25, 16:02:11 GMT+0 - Identified - Performance mitigations have been deployed to Railway.com, for any issues related to application connectivity, please raise a request on station.railway.com Mar 25, 16:06:10 GMT+0 - Monitoring - Railway.com load times are now back to normal. Mar 25, 16:06:31 GMT+0 - Resolved - Railway.com load times are now back to normal.

Type: Incident Duration: 3 hours and 55 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), Southeast Asia (Metal / Singapore), , US East (Metal / Virginia, USA), US West (Metal / California, USA), Edge Network (Railway Metal) → Mar 24, 23:50:00 GMT+0 - Investigating - We are aware of an issue affecting public traffic routing. Users may experience request timeouts or "Service Unavailable" errors when accessing their services. We are actively investigating and will provide updates as we learn more. Mar 25, 03:10:33 GMT+0 - Identified - We have identified the root cause and are working on a fix. Users may continue to experience request timeouts or "Service Unavailable" responses when accessing their services. We will provide updates as we make progress. Mar 25, 03:20:26 GMT+0 - Monitoring - A fix has been deployed, and we are monitoring for recovery. Users should see request timeouts and "Service Unavailable" responses subsiding. We will continue to monitor and provide a final update once we confirm stability. Mar 25, 03:45:02 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 16 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands) Mar 24, 18:54:31 GMT+0 - Investigating - We are currently investigating this incident. Mar 24, 19:05:24 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 24, 19:10:41 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 38 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands) Mar 23, 23:15:00 GMT+0 - Investigating - We are aware of issues affecting public networking in the EU West (Amsterdam) region. Users may experience connection failures or intermittent errors when accessing their services in this region. We are actively investigating and will provide updates as we learn more. Mar 23, 23:32:08 GMT+0 - Monitoring - We have identified the source of the connectivity issues in EU West (Amsterdam) and have applied a fix. We are now monitoring to confirm stability. Mar 23, 23:52:33 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 37 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), TCP Proxy (Metal), Southeast Asia (Metal / Singapore), , US East (Metal / Virginia, USA), US West (Metal / California, USA), Edge Network (Railway Metal) → Mar 21, 19:18:00 GMT+0 - Identified - We are aware of DNS resolution failures affecting Railway services. Users may experience failures when accessing their services via Railway-provided domains. Mar 21, 19:44:36 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 21, 19:55:11 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 10 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), , Build Machines (Metal), Image Registry (Metal), Builds (Railway Metal) → Mar 20, 14:12:43 GMT+0 - Investigating - We are investigating reports of degraded performance for services in EU West. Some users may experience slow builds, delayed container creation, or elevated response times. Our team is actively investigating. Mar 20, 14:49:35 GMT+0 - Identified - We identified the source of degraded performance in EU West and are continuing to work on a fix for this incident. Mar 20, 15:22:07 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 20, 16:22:43 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 38 minutes Affected Components: US East (Metal / Virginia, USA) Mar 19, 03:30:47 GMT+0 - Investigating - We are currently investigating this incident. Mar 19, 04:08:47 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 19, 05:09:16 GMT+0 - Resolved - This incident has been resolved. We found the issue pertaining to a single host in our fleet with elevated connections exhausting access in both the private and public network.

Type: Incident Duration: 1 hour and 43 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands) Mar 17, 10:57:35 GMT+0 - Investigating - We are aware of an issue affecting a subset of services hosted in EU West. Some users may experience their services being unavailable or deployments failing to start. This is caused by a suspected hardware failure impacting a portion of our compute infrastructure in this region. We are actively working on recovery and services are beginning to come back online. Services with attached storage may take longer to fully recover. Mar 17, 11:26:26 GMT+0 - Investigating - We are currently investigating this incident. If your service does not have a volume attached, please try re-deploying it Mar 17, 12:20:11 GMT+0 - Identified - We have identified the issue as a hardware failure on a single host in EU West. The affected infrastructure is being brought back online and workloads are recovering, some services have already been restored. Services with databases and attached storage may take a bit longer to fully come back. We will continue to provide updates as recovery progresses. Mar 17, 12:34:08 GMT+0 - Monitoring - All affected infrastructure is progressively recovering. If your service has not automatically recovered, please try redeploying. We are continuing to monitor. Mar 17, 12:40:21 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 5 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Deployments (Railway Metal) → Mar 16, 07:34:37 GMT+0 - Investigating - We are currently investigating this incident. Egress traffic may be failing on certain hosts from 0620 UTC, primarily concentrated in Asia region. Mar 16, 07:39:50 GMT+0 - Resolved - This incident has been resolved. We identified a subset of misconfigured hosts primarily in Asia that were failing to route outbound traffic correctly during 0620 UTC \~ 0735 UTC and have applied a fix. We are adding additional monitoring to detect this class of issue earlier. We apologize for any disruption. If you continue to experience issues, please contact support at .

Type: Incident Duration: 37 minutes Affected Components: DNS, EU West (Metal / Amsterdam, Netherlands), Traffic, US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , , Private Networking → Deployments (Railway Metal) → Mar 15, 09:27:36 GMT+0 - Investigating - We are currently investigating failures to reach the internet for a subset of deployments Mar 15, 09:32:21 GMT+0 - Investigating - We were able to confirm impact to be limited to part of the fleet, but to be affecting all networking for affected deployments. Mar 15, 09:42:15 GMT+0 - Identified - We have identified the root cause and are working on a fix Mar 15, 09:52:14 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 15, 10:04:12 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 6 minutes Affected Components: Logs Mar 13, 15:25:59 GMT+0 - Investigating - We're investigating an issue causing logs not to load. Mar 13, 15:39:51 GMT+0 - Identified - We've confirmed logs are loading with elevated delay. Our team is aware and working on a fix. Mar 13, 15:59:50 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 13, 16:50:20 GMT+0 - Identified - We're seeing continued degradation on logs loading time. We are continuing to work on a fix. Mar 13, 17:31:34 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 29 minutes Affected Components: , Build Machines (Metal), Image Registry (Metal), Builds (Railway Metal) → Mar 12, 20:29:16 GMT+0 - Investigating - We are currently investigating this incident. Non metal builders do not encounter this issue. Mar 12, 20:49:15 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 12, 20:58:18 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 9 minutes Affected Components: Build Machines (Metal) Mar 12, 08:30:00 GMT+0 - Investigating - We're investigating an issue causing certain build configurations (primarily for Dockerfile builds) to not resolve properly, causing builds to fail. Mar 12, 08:42:11 GMT+0 - Identified - We've identified the root cause and are working on a fix. Mar 12, 09:38:54 GMT+0 - Resolved - This incident has been resolved.

Type: Maintenance Duration: 2 hours and 12 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Deployments (Railway Metal) → Mar 11, 11:00:00 GMT+0 - Identified - Background: Recent growth in platform usage (9,000+ new users daily) has resulted in temporary queue overload during peak hours on two occasions. While Pro and Enterprise customers maintained deployment capability during these incidents, Trial and Hobby tier users experienced delays in build deployments. To prevent future occurrences and improve overall platform stability, we are implementing infrastructure improvements to our queue system. Expected Impact: We do not anticipate any service interruption during this maintenance window. However, out of an abundance of caution, please be aware that if any impact is observed, the following asynchronous actions may experience delays: \- Build deployments \- Staged changes \- Platform configuration updates Mitigation: This maintenance is scheduled during off-peak hours (11 AM – 2 PM GMT) to minimize user impact. We will monitor the upgrade closely and provide real-time updates if any issues arise. Next Steps: We will post a follow-up update upon completion of this maintenance window. Thank you for your patience as we continue to improve the Railway platform. For more information, or questions please see: Mar 11, 11:03:12 GMT+0 - Identified - Maintenance is now in progress. Mar 11, 11:48:20 GMT+0 - Identified - Phase 1 has completed successfully. We are moving on with Phase 2, and will report any downtime here. We expect operations to keep running smoothly. Mar 11, 12:08:10 GMT+0 - Identified - We are moving on to Phase 3\. There may be some impact to builds, which we will call here if needed. Mar 11, 12:49:40 GMT+0 - Identified - We've completed all phases of our maintenance, and are monitoring for any impact before closing out. Trial users may see their deploy queued for a few minutes Mar 11, 13:15:22 GMT+0 - Completed - Maintenance has completed successfully.

Type: Incident Duration: 5 hours and 31 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Deployments (Railway Metal) → Mar 10, 15:19:01 GMT+0 - Investigating - We are currently investigating an issue causing deployments to take longer than normal Mar 10, 15:36:39 GMT+0 - Identified - We've pushed out a fix and are seeing slow recovery. Mar 10, 15:59:40 GMT+0 - Monitoring - We're seeing recovery, with delays primarily impacting trial deployments. We are monitoring for full recovery. Mar 10, 16:40:23 GMT+0 - Monitoring - We are continuing to monitor this incident. Deployments may be queued, but are going through Mar 10, 17:20:20 GMT+0 - Monitoring - Our team continues to work on this incident. Deploys are still slow, but not failing. Mar 10, 18:00:15 GMT+0 - Monitoring - Deployment slowness is persisting. Deploys are completing but with longer than normal queue and build times. Our team is actively working on a resolution. Mar 10, 18:58:28 GMT+0 - Monitoring - Elevated load continues to cause slowness on our system. We're pausing Free, Trial and Hobby deployments to assist with deployment backpressure. Mar 10, 19:23:18 GMT+0 - Monitoring - Elevated load continues to cause slowness on our system. We' have restored all Pro deployments back to nominal, we are turning on Hobby slowly to assist with deployment backpressure. Mar 10, 19:51:00 GMT+0 - Monitoring - We've successfully eliminated slowness from our deployment pipeline, and are re-enabling deployments for all plans as the backpressure is recovering. Mar 10, 20:07:14 GMT+0 - Monitoring - We are seeing improvement — deployments are now processing without being queued. We continue to monitor the situation and will provide a further update once we are confident the issue is fully resolved Mar 10, 20:49:54 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 22 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Build Machines (Metal), Image Registry (Metal), , Builds (Railway Metal) → Deployments (Railway Metal) → Mar 9, 22:16:39 GMT+0 - Investigating - We are aware of an issue where new builds may appear stuck and fail to complete or progress slowly. Existing running services are not affected. We are actively investigating and working on a fix. Mar 9, 22:56:23 GMT+0 - Identified - We have identified the root cause and are deploying a fix. Mar 9, 22:26:24 GMT+0 - Investigating - Hobby, free and trial deploys have been temporarily disabled. We are continuing our investigation into the root issue Mar 9, 23:35:06 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 9, 23:38:27 GMT+0 - Resolved - This incident has been resolved. New deployments for free, trial and hobby have been re-enabled.

Type: Incident Duration: 45 minutes Affected Components: TCP Proxy (Metal) Mar 9, 16:00:00 GMT+0 - Investigating - We are working on a fix for TCP proxy connection issues, primarily in the US East and Asia regions Mar 9, 16:23:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 9, 16:25:07 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 9, 16:45:15 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 7 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Deployments (Railway Metal) → Mar 9, 15:26:35 GMT+0 - Investigating - We're investigating an issue causing a large portion of deployments to error, causing slowdowns across deployments and builds. Mar 9, 15:30:49 GMT+0 - Identified - We have identified the root cause of the issue and are working on a fix. Mar 9, 15:47:36 GMT+0 - Identified - We are continuing to work on a fix for this incident. New deploys for the hobby, free and trial plans have been paused until a fix is deployed. Mar 9, 16:04:43 GMT+0 - Identified - A fix has been pushed and is pending rollout. We're seeing issues with TCP proxy connections, related to this incident. Mar 9, 16:24:41 GMT+0 - Monitoring - We are seeing recovery for deployments. We are continuing to work on a fix for the TCP Proxy: Mar 9, 16:26:05 GMT+0 - Monitoring - We are seeing recovery, and have re-enabled Free, Trial and Hobby deploys. Mar 9, 16:33:18 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Mar 6, 14:43:00 GMT+0 - Resolved - Between 14:43 UTC and 14:50 UTC, users experienced HTTP 503 errors when accessing [railway.com](http://railway.com) and Railway-hosted services. We identified the root cause as our CDN provider (Fastly) reaching maximum connection limits to our edge network, which resulted in 503 errors for end users. The issue has been resolved and all services are operating normally. We are working with our CDN provider to implement safeguards to prevent this from recurring. We apologize for the disruption. If you continue to experience issues, please contact support at .

Type: Incident Duration: 16 minutes Affected Components: Dashboard (railway.com) Mar 5, 16:41:35 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 5, 16:49:29 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 5, 16:57:56 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 15 minutes Affected Components: Southeast Asia (Metal / Singapore), EU West (Metal / Amsterdam, Netherlands), TCP Proxy (Metal), , US East (Metal / Virginia, USA), US West (Metal / California, USA), Edge Network (Railway Metal) → Mar 5, 09:36:58 GMT+0 - Identified - We are working on a fix for network degradation on the railway edge. Mar 5, 09:50:20 GMT+0 - Monitoring - We've applied a fix and are seeing recovery across all regions. Mar 5, 09:52:00 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 18 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Build Machines (Metal), Image Registry (Metal), , Builds (Railway Metal) → Deployments (Railway Metal) → Mar 3, 23:05:00 GMT+0 - Investigating - We're investigating an issue causing slowdowns in builds. Mar 3, 23:27:56 GMT+0 - Investigating - We've paused free, trial and hobby plan deploys as we continue investigating the issue. Mar 3, 23:32:07 GMT+0 - Identified - We've identified the root cause of the issue and are working on a fix. Free, Trial and Hobby provisions remain disabled at this time. Mar 3, 23:51:01 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 3, 23:52:22 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 3, 23:59:23 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Non-pro deploys remain disabled until we catch up with the deploy queue. Mar 4, 00:16:17 GMT+0 - Monitoring - Hobby, free and trial deployments have been re-enabled. We are still monitoring, but have seen full recovery on our deploy systems. Mar 4, 00:23:10 GMT+0 - Resolved - This incident has been resolved.