Flightcontrol Outage History

Some machines in SIN are unreachable. A few Managed Postgres clusters may fail to fail-over or update. We are in the process of fixing this with our upstream provider.

We are seeing recovery of network connectivity between SIN and most destinations. We're continuing to work with our upstream provider to resolve the remaining issues.

We are seeing a recurrence of elevated latency across some hosts in ORD impacting the MPG control plane and a subset of clusters there. We are working to address this.

We are working with our upstream network provider to address periodic loss of connectivity over transit in ord

Some of our edge nodes in GRU has suffered an error that crashed some of the critical services. We're currently working to bring them back online. Some traffic entering through GRU (i.e. users connecting from around GRU) may be temporarily affected: connections may see increased latency or be occasionally dropped.

We're performing an emergency maintenance on Petsem, our secrets management service. Some control plane write operations may temporarily fail, for example, creating new apps or secrets. Existing apps and machines should keep functioning without issues.

We are continuing to address this issue. Some clusters in IAD are unavailable at this time, some users may have seen unexpected cluter restarts. We are working on restoring normal performance for all clusters in IAD

One of our Consul clusters is in degraded state due to a failed node. This can cause issues with LiteFS primary node selection, Unmanaged Postgres (14.x and older *only*), and creation of new Unmanaged Postgres clusters. Impact is limited to these legacy products and does not affect deployments, running Fly applications in general, or Managed Postgres clusters.

Due to an upstream provider issue, Private Networking (6PN) is currently degraded in SYD region. Communication between Machines in SYD region and Machines in other regions may fail at this time. Newly created Machines in SYD may fail to sync to other regions (may not show up in Machines API List endpoint, or state may be incorrect). Additionally, TLS certificate resolution and Machines API authentication may currently be degraded in the SYD region. We are working with our upstream providers to resolve this issue.

We identified the issue and are working on a fix.

We are aware of increased latency and connection drops for clients located near Chicago (ORD) and are currently working on a fix.

We're still seeing connection issues originating from SJC/LAX/West Coast US and are still investigating.

The EWR region is low on available host capacity. Creating new machines, or starting currently stopped/suspended machines, may fail at this time. We are working on provisioning new host capacity in the region. Please consider using nearby regions such as ord if possible.

One of our upstreams is experiencing high packet loss and latency. We are actively working with them.

All MPGs in SIN are reachable again. We are seeing high latency to the affected provider. This may affect a subset of machines hosted in SIN.

We've identified the issue and are working on a fix.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.