Railway Outage History

Type: Incident Duration: 1 hour and 24 minutes Affected Components: US East (Metal / Virginia, USA) Mar 25, 20:23:24 GMT+0 - Investigating - We are currently investigating this incident. Mar 25, 21:17:27 GMT+0 - Identified - We have traced this back to a change rolled out with static IP egress gateways. We have begun the rollbacks and are already seeing recovery. Mar 25, 21:29:49 GMT+0 - Monitoring - The rollback has completed, we are seeing recovery, and are currently monitoring the result. Mar 25, 21:47:30 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 15 minutes Affected Components: , External / Third-Party Integrations → Mar 25, 18:05:54 GMT+0 - Investigating - Dockerhub Registry Outage, pulls from Dockerhub may fail. Mar 25, 19:21:23 GMT+0 - Resolved - Dockerhub reports recovery

Type: Incident Duration: 1 hour and 45 minutes Affected Components: , Edge Network (Legacy) → Mar 25, 16:09:16 GMT+0 - Investigating - Cert discrepancy between Fastly and Railway leading to SAN errors. The network team is actively triaging the issue for affected customers. Mar 25, 16:11:10 GMT+0 - Identified - For some affected domains, cert provisioning is inoperable, this is due to incorrect domain configuration, please re-add your domains to triage the issue. Mar 25, 16:38:28 GMT+0 - Identified - Manual workarounds for customers show that access is restored, we are working with our upstream vendor to implement domain forwarding to mitigate customer misconfiguration. Mar 25, 17:54:18 GMT+0 - Resolved - Network behavior is back to nominal.

Type: Incident Duration: 12 minutes Affected Components: Dashboard (railway.com) Mar 25, 15:55:01 GMT+0 - Investigating - Railway Dashboard Load Times Impacted Mar 25, 16:02:11 GMT+0 - Identified - Performance mitigations have been deployed to Railway.com, for any issues related to application connectivity, please raise a request on station.railway.com Mar 25, 16:06:10 GMT+0 - Monitoring - Railway.com load times are now back to normal. Mar 25, 16:06:31 GMT+0 - Resolved - Railway.com load times are now back to normal.

Type: Incident Duration: 16 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands) Mar 24, 18:54:31 GMT+0 - Investigating - We are currently investigating this incident. Mar 24, 19:05:24 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 24, 19:10:41 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 38 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands) Mar 23, 23:15:00 GMT+0 - Investigating - We are aware of issues affecting public networking in the EU West (Amsterdam) region. Users may experience connection failures or intermittent errors when accessing their services in this region. We are actively investigating and will provide updates as we learn more. Mar 23, 23:32:08 GMT+0 - Monitoring - We have identified the source of the connectivity issues in EU West (Amsterdam) and have applied a fix. We are now monitoring to confirm stability. Mar 23, 23:52:33 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 37 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), TCP Proxy (Metal), Southeast Asia (Metal / Singapore), , US East (Metal / Virginia, USA), US West (Metal / California, USA), Edge Network (Railway Metal) → Mar 21, 19:18:00 GMT+0 - Identified - We are aware of DNS resolution failures affecting Railway services. Users may experience failures when accessing their services via Railway-provided domains. Mar 21, 19:44:36 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 21, 19:55:11 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 10 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), , Build Machines (Metal), Image Registry (Metal), Builds (Railway Metal) → Mar 20, 14:12:43 GMT+0 - Investigating - We are investigating reports of degraded performance for services in EU West. Some users may experience slow builds, delayed container creation, or elevated response times. Our team is actively investigating. Mar 20, 14:49:35 GMT+0 - Identified - We identified the source of degraded performance in EU West and are continuing to work on a fix for this incident. Mar 20, 15:22:07 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 20, 16:22:43 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 38 minutes Affected Components: US East (Metal / Virginia, USA) Mar 19, 03:30:47 GMT+0 - Investigating - We are currently investigating this incident. Mar 19, 04:08:47 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 19, 05:09:16 GMT+0 - Resolved - This incident has been resolved. We found the issue pertaining to a single host in our fleet with elevated connections exhausting access in both the private and public network.

Type: Incident Duration: 5 minutes Affected Components: EU West (Metal / Amsterdam, Netherlands), US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , Deployments (Railway Metal) → Mar 16, 07:34:37 GMT+0 - Investigating - We are currently investigating this incident. Egress traffic may be failing on certain hosts from 0620 UTC, primarily concentrated in Asia region. Mar 16, 07:39:50 GMT+0 - Resolved - This incident has been resolved. We identified a subset of misconfigured hosts primarily in Asia that were failing to route outbound traffic correctly during 0620 UTC \~ 0735 UTC and have applied a fix. We are adding additional monitoring to detect this class of issue earlier. We apologize for any disruption. If you continue to experience issues, please contact support at .

Type: Incident Duration: 37 minutes Affected Components: DNS, EU West (Metal / Amsterdam, Netherlands), Traffic, US East (Metal / Virginia, USA), Southeast Asia (Metal / SIngapore), US West (Metal / California, USA), , , Private Networking → Deployments (Railway Metal) → Mar 15, 09:27:36 GMT+0 - Investigating - We are currently investigating failures to reach the internet for a subset of deployments Mar 15, 09:32:21 GMT+0 - Investigating - We were able to confirm impact to be limited to part of the fleet, but to be affecting all networking for affected deployments. Mar 15, 09:42:15 GMT+0 - Identified - We have identified the root cause and are working on a fix Mar 15, 09:52:14 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 15, 10:04:12 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 6 minutes Affected Components: Logs Mar 13, 15:25:59 GMT+0 - Investigating - We're investigating an issue causing logs not to load. Mar 13, 15:39:51 GMT+0 - Identified - We've confirmed logs are loading with elevated delay. Our team is aware and working on a fix. Mar 13, 15:59:50 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 13, 16:50:20 GMT+0 - Identified - We're seeing continued degradation on logs loading time. We are continuing to work on a fix. Mar 13, 17:31:34 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 29 minutes Affected Components: , Build Machines (Metal), Image Registry (Metal), Builds (Railway Metal) → Mar 12, 20:29:16 GMT+0 - Investigating - We are currently investigating this incident. Non metal builders do not encounter this issue. Mar 12, 20:49:15 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 12, 20:58:18 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 9 minutes Affected Components: Build Machines (Metal) Mar 12, 08:30:00 GMT+0 - Investigating - We're investigating an issue causing certain build configurations (primarily for Dockerfile builds) to not resolve properly, causing builds to fail. Mar 12, 08:42:11 GMT+0 - Identified - We've identified the root cause and are working on a fix. Mar 12, 09:38:54 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 45 minutes Affected Components: TCP Proxy (Metal) Mar 9, 16:00:00 GMT+0 - Investigating - We are working on a fix for TCP proxy connection issues, primarily in the US East and Asia regions Mar 9, 16:23:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 9, 16:25:07 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 9, 16:45:15 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Mar 6, 14:43:00 GMT+0 - Resolved - Between 14:43 UTC and 14:50 UTC, users experienced HTTP 503 errors when accessing [railway.com](http://railway.com) and Railway-hosted services. We identified the root cause as our CDN provider (Fastly) reaching maximum connection limits to our edge network, which resulted in 503 errors for end users. The issue has been resolved and all services are operating normally. We are working with our CDN provider to implement safeguards to prevent this from recurring. We apologize for the disruption. If you continue to experience issues, please contact support at .

Type: Incident Duration: 16 minutes Affected Components: Dashboard (railway.com) Mar 5, 16:41:35 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 5, 16:49:29 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 5, 16:57:56 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 15 minutes Affected Components: Southeast Asia (Metal / Singapore), EU West (Metal / Amsterdam, Netherlands), TCP Proxy (Metal), , US East (Metal / Virginia, USA), US West (Metal / California, USA), Edge Network (Railway Metal) → Mar 5, 09:36:58 GMT+0 - Identified - We are working on a fix for network degradation on the railway edge. Mar 5, 09:50:20 GMT+0 - Monitoring - We've applied a fix and are seeing recovery across all regions. Mar 5, 09:52:00 GMT+0 - Resolved - This incident has been resolved.