Fly.io Outage History
Fly.io is up right nowPingoru has recorded 196 incidents for Fly.io: 196 in the last 12 months and 76 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 13 | 79 of 720 (11%) |
| May 2026 | 29 | 114 of 744 (15%) |
| June 2026 | 22 | 100 of 720 (14%) |
| July 2026 | 18 | 34 of 744 (5%) |
| August 2026 | 33 | 34 of 744 (5%) |
| September 2026 | 26 | 25 of 720 (3%) |
| October 2026 | 14 | 30 of 591 (5%) |
Incidents, newest first
Minor March 27, 2026 Low capacity in IAD 3h 42m Deployments, IAD - Ashburn, Virginia (US)
Started Mar 27, 2026, 06:08 PM UTC· resolved Mar 27, 2026, 09:51 PM UTC
- investigating Mar 27, 2026, 06:08 PM UTC
We're currently investigating capacity issues in IAD that is preventing machine starts (machine creates are currently unaffected). This may result in deploys failing to complete (even for apps outside of the IAD region). As a workaround, using legacy Fly builders explicitly located in another region (i.e., `FLY_REMOTE_BUILDER_REGION=lhr fly deploy --depot=false --recreate-builder`) may help in the meantime.
- investigating Mar 27, 2026, 06:47 PM UTC
We're continuing to evaluate our options for increasing short-term capacity in the IAD region.
- identified Mar 27, 2026, 07:21 PM UTC
We've brought some additional capacity online in IAD and are seeing improvements, and we're continuing to work on adding more and freeing up additional room.
- monitoring Mar 27, 2026, 09:09 PM UTC
With the additional capacity we've brought online, machine start failure rates in IAD have now recovered. We'll continue to monitor IAD capacity.
- resolved Mar 27, 2026, 09:51 PM UTC
This incident has been resolved.
Major March 26, 2026 Machine Creates Failing in ORD Region 2h 33m ORD - Chicago, Illinois (US)
Started Mar 26, 2026, 03:21 PM UTC· resolved Mar 26, 2026, 05:54 PM UTC
- investigating Mar 26, 2026, 03:21 PM UTC
We are currently investigating elevated errors creating machines in the ORD (Chicago, Illinois) region. Users may see `failed to launch VM: request returned non-2xx status: 408` errors when creating, updating, or scaling machines in ORD. Existing, already running machines in the ORD region continue to run as normal.
- investigating Mar 26, 2026, 04:08 PM UTC
We are continuing to investigate this issue. We are seeing 408 errors decreasing in ORD, though still above baseline.
- identified Mar 26, 2026, 04:50 PM UTC
We've identified the cause of this increased failure rate and a fix is in progress. We are seeing most creates in ORD succeed at this time, though failure rate is still above baseline.
- monitoring Mar 26, 2026, 05:28 PM UTC
We've implemented a fix and have seen error rates for machine creates in ORD drop off. We're continuing to monitor the results.
- resolved Mar 26, 2026, 05:54 PM UTC
This incident has been resolved.
Critical March 26, 2026 Network issues in FRA region 1h 42m Management Plane - FRA, FRA - Frankfurt, Germany
Started Mar 26, 2026, 12:37 PM UTC· resolved Mar 26, 2026, 02:19 PM UTC
- investigating Mar 26, 2026, 12:37 PM UTC
We are investigating network issues in FRA region. Apps and/or Managed Postgres clusters in the region may be inaccessible at this time.
- monitoring Mar 26, 2026, 01:14 PM UTC
Apps and Managed Postgres clusters in FRA region should be back online at this time. We are monitoring for any further issues.
- identified Mar 26, 2026, 01:16 PM UTC
Some Managed Postgres clusters in FRA region are still unreachable, we are investigating this issue.
- resolved Mar 26, 2026, 02:19 PM UTC
This incident has been resolved.
Notice March 23, 2026 Backend errors when trying to use Grafana to view logs 1h 9m Logs
Started Mar 23, 2026, 03:18 PM UTC· resolved Mar 23, 2026, 04:27 PM UTC
- investigating Mar 23, 2026, 03:18 PM UTC
Using the Logs panel in Grafana at https://fly-metrics.net/ will show a 502 error from the backend and won't show any logs. You can use `fly logs` or the live log viewer directly on https://fly.io/dashboard to view streaming logs for the time being.
- identified Mar 23, 2026, 03:41 PM UTC
Using the Logs panel in Grafana at https://fly-metrics.net/ will show a 502 error from the backend and won't show any logs. You can use `fly logs` or the live log viewer directly on https://fly.io/dashboard to view streaming logs for the time being.
- monitoring Mar 23, 2026, 03:55 PM UTC
We've deployed a fix and are monitoring the results. Logs are now be visible on Grafana.
- resolved Mar 23, 2026, 04:27 PM UTC
This incident is resolved, Grafana logs are now working properly.
Minor March 20, 2026 Machines failing to start in DFW 3d 5h DFW - Dallas, Texas (US)
Started Mar 20, 2026, 07:26 AM UTC· resolved Mar 23, 2026, 01:19 PM UTC
- investigating Mar 20, 2026, 07:26 AM UTC
The Machines start failure rate is elevated in DFW.
- monitoring Mar 20, 2026, 08:08 AM UTC
We freed up some capacity on our workers to allow for successful Machine starts.
- monitoring Mar 20, 2026, 12:45 PM UTC
In addition to freeing up existing capacity, the team has provisioned new capacity in DFW and we are monitoring the results.
- monitoring Mar 21, 2026, 08:26 AM UTC
Machine start success rates in DFW have improved but we are continuing to monitor and make further adjustments. We will provide updates as the situation progresses.
- resolved Mar 23, 2026, 01:19 PM UTC
This incident has been resolved.
Major March 18, 2026 IPv6 networking issues in SJC region 49m SJC - San Jose, California (US)
Started Mar 18, 2026, 04:12 PM UTC· resolved Mar 18, 2026, 05:02 PM UTC
- investigating Mar 18, 2026, 04:12 PM UTC
We are investigating intermittent network issues in SJC region impacting outbound public IPv6 access from Machines. Connecting to IPv6 internet resources from apps hosted in SJC region may be slow or fail at this time. IPv4 access, as well as 6PN private networking, are unaffected.
- monitoring Mar 18, 2026, 04:31 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 18, 2026, 05:02 PM UTC
This incident has been resolved.
Minor March 18, 2026 Fly ssh console command failing 5m Machines API
Started Mar 18, 2026, 02:12 PM UTC· resolved Mar 18, 2026, 02:18 PM UTC
- identified Mar 18, 2026, 02:12 PM UTC
We have identified an issue causing new `fly ssh console` connections to fail with 500 errors. A fix is in progress.
- monitoring Mar 18, 2026, 02:17 PM UTC
A fix has been implemented and we are seeing `ssh console` commands succeed as normal.
- resolved Mar 18, 2026, 02:18 PM UTC
This incident has been resolved.
Minor March 18, 2026 Connection Issues in SJC 11m Management Plane - SJC, SJC - San Jose, California (US)
Started Mar 18, 2026, 02:07 PM UTC· resolved Mar 18, 2026, 02:18 PM UTC
- monitoring Mar 18, 2026, 02:07 PM UTC
Between 13:55 and 14:03 UTC machines and MPG clusters hosted in the SJC region saw elevated connection errors. Users may have seen errors connecting to or from most machines in the region, as well as with deployments or updates to machines in the region. Networking has returned to normal in the region, and we are continuing to monitor closely to ensure stable recovery.
- resolved Mar 18, 2026, 02:18 PM UTC
This incident has been resolved.
Notice March 14, 2026 Sprites Operations: 401 errors for certain organizations 9h 44m Sprites
Started Mar 14, 2026, 04:20 AM UTC· resolved Mar 14, 2026, 02:05 PM UTC
- monitoring Mar 14, 2026, 01:55 PM UTC
Organizations with names prefixed with numerical digits may experience 401 errors. Affected operations include actions such as Sprite creation, listing, etc... A fix has been implemented since 2026-03-14 12:30 UTC and we are monitoring the results!
- resolved Mar 14, 2026, 02:05 PM UTC
This incident has been resolved.
Major March 11, 2026 Setting secrets and creating apps is degraded 33m Machines API, Deployments
Started Mar 11, 2026, 11:03 AM UTC· resolved Mar 11, 2026, 11:37 AM UTC
- identified Mar 11, 2026, 09:19 AM UTC
An ongoing data migration in our secret storage service is causing degraded Machines API functionality.
- monitoring Mar 11, 2026, 10:14 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Mar 11, 2026, 11:03 AM UTC
While the secret storage service was in a read-only state, app creation requests queued up, due to the retry logic and insufficient request concurrency limits in our GraphQL API. This prevented our GraphQL API from serving any other requests. We have scaled up the GraphQL API and are continuing to monitor the situation.
- resolved Mar 11, 2026, 11:37 AM UTC
This incident has been resolved.
Major March 7, 2026 Private networking issues in SYD region 1h 14m SYD - Sydney, Australia
Started Mar 07, 2026, 02:42 PM UTC· resolved Mar 07, 2026, 03:56 PM UTC
- investigating Mar 07, 2026, 02:42 PM UTC
We are investigating a private networking failure between SYD and other regions. Apps continue to run, and private networking within SYD is unaffected.
- monitoring Mar 07, 2026, 03:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 07, 2026, 03:56 PM UTC
This incident has been resolved.
Notice March 5, 2026 Routing issues in NA regions 26m
Started Mar 05, 2026, 07:24 PM UTC· resolved Mar 05, 2026, 07:50 PM UTC
- investigating Mar 05, 2026, 07:24 PM UTC
We're aware of routing issues affecting some customers in North America regions, and we're actively investigating.
- monitoring Mar 05, 2026, 07:38 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 05, 2026, 07:50 PM UTC
This incident has been resolved. Due to a BGP issue, we saw some North American traffic routed to edges in Singapore (sin). Users in North America would have seen additional request latency during this period.
Major March 3, 2026 Elevated GraphQL API errors 57m Dashboard, Deployments
Started Mar 03, 2026, 08:18 PM UTC· resolved Mar 03, 2026, 09:15 PM UTC
- investigating Mar 03, 2026, 08:18 PM UTC
We're investigating elevated GraphQL errors that affect some API endpoints.
- monitoring Mar 03, 2026, 08:36 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 03, 2026, 09:15 PM UTC
This incident was caused by a failed Redis node that powers our GraphQL API. We were able to recreate the Redis node and restore service. We are still investigating the root cause of the failure. In the mean time, all API endpoints now appear to be stable and errors have dropped to baseline level.
Minor March 3, 2026 Cost Explorer fails to load 1h 20m Dashboard
Started Mar 03, 2026, 10:50 AM UTC· resolved Mar 03, 2026, 12:10 PM UTC
- investigating Mar 03, 2026, 10:50 AM UTC
We are currently investigating this issue. The page currently displays: "We’re having trouble loading the cost breakdown."
- resolved Mar 03, 2026, 12:10 PM UTC
This incident has been resolved.
Notice March 3, 2026 Certificates issues affecting API and proxy –
Started Mar 03, 2026, 12:54 AM UTC· resolved Mar 03, 2026, 12:54 AM UTC
- resolved Mar 03, 2026, 02:05 AM UTC
Between 19:54 and 20:06 UTC, our Vault cluster serving app certificates was unavailable. This caused various API requests to fail, mainly operations on certificates but also app creates and IP assignments. As the failure mode was Vault requests hanging rather than failing immediately, TLS requests through fly-proxy for domains where the certificate was not cached on the local node remained open for a long time while proxy attempted to fetch the certificate; this caused some connections to fail as too many connection slots were taken up by requests waiting on Vault. The root cause of this incident was a partially completed update to the Vault cluster. We will be implementing safeguards in the proxy for this failure mode, as well as improving certificate storage longer-term.
Minor March 2, 2026 Issues with the Machines API 30m Dashboard, Machines API, Deployments
Started Mar 02, 2026, 09:19 PM UTC· resolved Mar 02, 2026, 09:50 PM UTC
- investigating Mar 02, 2026, 09:19 PM UTC
We're currently investigating issues with the Machines API. Customer deployments and the Fly dashboard may be affected.
- identified Mar 02, 2026, 09:39 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 02, 2026, 09:47 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 02, 2026, 09:50 PM UTC
This incident has been resolved.
Major March 2, 2026 Machines failing to boot in EWR 5h 7m EWR - Secaucus, NJ (US)
Started Mar 02, 2026, 05:42 PM UTC· resolved Mar 02, 2026, 10:49 PM UTC
- investigating Mar 02, 2026, 05:42 PM UTC
We are currently investigating this issue.
- identified Mar 02, 2026, 06:21 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 02, 2026, 08:35 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 02, 2026, 10:49 PM UTC
This incident has been resolved.
Major February 27, 2026 Slow API requests 1h 31m Dashboard, Machines API, Deployments, Remote Builds and 1 more
Started Feb 27, 2026, 06:50 PM UTC· resolved Feb 27, 2026, 08:21 PM UTC
- investigating Feb 27, 2026, 06:50 PM UTC
We are investigating increased in API request latency and timeouts with the main platform API. This is impacting multiple operations, including creating, querying or performing actions against machines, as well as platform level operations like adding payment methods.
- identified Feb 27, 2026, 06:52 PM UTC
We have identified the cause of the increased latency and are working on a fix. The most common errors we are seeing is timeouts when users attempt to perform an action against a newly created app / machine resource. Those may timeout or fail with an `app|machine not found` error
- identified Feb 27, 2026, 06:53 PM UTC
We are continuing to work on a fix for this issue.
- identified Feb 27, 2026, 06:59 PM UTC
New Sprite creations are also timing out or failing at this time. We are continuing to work on a fix for this issue.
- identified Feb 27, 2026, 07:05 PM UTC
We are currently seeing full API failures for requests to our Graphql API and elevated failures for the machines API. Direct calls to these apis may fail, along with many flyctl commands. We have identified the cause of the issue and are continuing to work on a fix. Existing running machines and apps should continue to be reachable, but creates, deploys, or other features relying on platform API calls will fail at this time.
- identified Feb 27, 2026, 07:23 PM UTC
An initial fix has been deployed and we are seeing improvements in load and API performance. Some operations that rely on the Graphql API, such as new app creations and some deployments, will continue to fail at this time. We are continuing to work on restoring full availability.
- identified Feb 27, 2026, 07:41 PM UTC
A second fix has been deployed and database load has returned to normal, resulting in API response times beginning to normalize. Most Machines API requests should succeed as normal, and deploys to existing apps should also work. We are working through a backlog of background jobs. New app / organization creations and other other operations that use these will continue to see increased latency or failures while we work thorough these. New MPG cluster and new Sprite creation continues to be impacted.
- monitoring Feb 27, 2026, 08:05 PM UTC
API and platform operations have normalized. We are continuing to monitor to ensure full and stable recovery. Background jobs are almost fully caught up. Users may still see slightly slower requests creating new apps / orgs, but they should complete successfully. Sprite and MPG cluster creations are processing as normal.
- resolved Feb 27, 2026, 08:21 PM UTC
This incident has been resolved. All platform and API operations are working normally.
Minor February 27, 2026 Capacity issues in iad and dfw 2h 20m Deployments
Started Feb 27, 2026, 03:34 PM UTC· resolved Feb 27, 2026, 05:54 PM UTC
- identified Feb 27, 2026, 03:34 PM UTC
These regions (Dallas, TX dfw and Ashburn, VA iad) are currently low on capacity. New machine creates in these regions might fail temporarily, and Depot builders may be unavailable, causing deploys to hang in "Waiting for Depot builder". If you are having issues with Depot builders, consider moving them to a different non-iad, non-dfw region in your fly.io dashboard's "Settings" page under "App builders", or try `--depot=false`.
- monitoring Feb 27, 2026, 05:31 PM UTC
We have provisioned additional capacity in dfw and iad and are monitoring to ensure machine and builder starts are succeeding consistently.
- resolved Feb 27, 2026, 05:54 PM UTC
This incident has been resolved.
Notice February 26, 2026 Capacity isssues in iad and dfw 5h 27m Management Plane - IAD, Remote Builds, DFW - Dallas, Texas (US), IAD - Ashburn, Virginia (US)
Started Feb 26, 2026, 05:00 PM UTC· resolved Feb 26, 2026, 10:28 PM UTC
- identified Feb 26, 2026, 05:00 PM UTC
We have identified the problem and are working on a fix.
- identified Feb 26, 2026, 05:05 PM UTC
New machine creates in these regions might fail temporarily, and Depot builders may be unavailable. If you are having issues with Depot builders, consider moving them to a different region, or try `--depot=false`.
- identified Feb 26, 2026, 05:18 PM UTC
We've identified some newly created Managed Postgres clusters are failing to come up healthy in these regions.
- identified Feb 26, 2026, 06:57 PM UTC
We have added additional capacity in DFW and IAD regions and are monitoring the impact. New machine creates and deploys without volumes are seeing improved success rates. Deploys using depot builders in those regions are also improving, with much quicker builder start times. Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use `fly volume fork --vm-memory ` to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.
- monitoring Feb 26, 2026, 08:19 PM UTC
We're continuing to monitor after having added more capacity to our DFW and IAD regions. Deploys or machine starts using existing volumes in these regions may still hit a capacity issue. Users should use `fly volume fork --vm-memory ` to fork the volume to a host with more capacity, then retry the deploy or start command using the new volume.
- resolved Feb 26, 2026, 10:28 PM UTC
This incident has been resolved.
Notice February 24, 2026 Sprites API degradation 28m Sprites
Started Feb 24, 2026, 05:23 PM UTC· resolved Feb 24, 2026, 05:51 PM UTC
- identified Feb 24, 2026, 05:23 PM UTC
A slow deploy is causing Sprites API degradation. We are implementing a fix.
- identified Feb 24, 2026, 05:24 PM UTC
A slow deploy is causing Sprites API degradation. We are implementing a fix.
- resolved Feb 24, 2026, 05:51 PM UTC
This incident has been resolved.
Minor February 24, 2026 Sprite creations failing 1h 4m Sprites
Started Feb 24, 2026, 09:39 AM UTC· resolved Feb 24, 2026, 10:44 AM UTC
- investigating Feb 24, 2026, 09:39 AM UTC
We are currently investigating issues creating new Sprites.
- monitoring Feb 24, 2026, 10:25 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 24, 2026, 10:44 AM UTC
This incident has been resolved.
Minor February 24, 2026 Metrics are degraded 6h 32m Metrics
Started Feb 24, 2026, 04:33 AM UTC· resolved Feb 24, 2026, 11:06 AM UTC
- identified Feb 24, 2026, 04:33 AM UTC
In some cases data is missing or lagging. We've identified the problem and are working on a fix.
- identified Feb 24, 2026, 05:49 AM UTC
We're continuing to work with VictoriaMetrics support on a fix for this issue.
- monitoring Feb 24, 2026, 06:46 AM UTC
Metrics are coming back online, but it will take a little time to process what's backed up in the queues.
- monitoring Feb 24, 2026, 09:35 AM UTC
Delayed metrics are still being processed.
- resolved Feb 24, 2026, 11:06 AM UTC
Metrics processing has caught up, and we don't see any data loss.
Notice February 23, 2026 Degraded Managed Postgres Control Plane – Management Plane - IAD, Management Plane - LHR, Management Plane - SIN
Started Feb 23, 2026, 08:30 PM UTC· resolved Feb 23, 2026, 08:30 PM UTC
- investigating Feb 23, 2026, 03:00 PM UTC
We are currently investigating issues with the MPG control plane. Users may experience delays or hanging when creating or deleting databases via the dashboard or CLI.
- resolved Feb 24, 2026, 12:31 AM UTC
This incident has been resolved as of 20:30 UTC.
Minor February 20, 2026 Deploys hanging at waiting for Depot Builder 4h 35m Deployments
Started Feb 20, 2026, 04:14 PM UTC· resolved Feb 20, 2026, 08:49 PM UTC
- monitoring Feb 20, 2026, 04:14 PM UTC
We have seen elevated latency provisioning Depot builders during deployments over the past hour. This caused some deploys to hang or timeout at the "Waiting for Depot Builder" step in this period. Latency has improved and builder provision times are back to normal. We're continuing to monitor to ensure latency remains normal.
- identified Feb 20, 2026, 04:39 PM UTC
We are again seeing elevated latency provisioning depot builders on new deploys. Users may see deploys using Depot builders hang or timeout at the "Waiting for Depot Builder" step. We are working on a fix. We are switching all deploys to use the default Fly builders in the meantime. If desired users can manually switch back to depot builders using `fly deploy --depot=true` but may continue to see latency issues at this time.
- identified Feb 20, 2026, 05:59 PM UTC
A fix is being rolled out. Fly builders continue to be the default while this is deployed
- monitoring Feb 20, 2026, 07:38 PM UTC
The fix has been rolled out and we are seeing deploys using depot builder succeeding normally. We continue to monitor to ensure full recovery. Depot builders have been reenabled as the default option for new deploys
- resolved Feb 20, 2026, 08:49 PM UTC
This incident has been resolved.