Flightcontrol incident

Macaroon Auth + Machines API Issues

Critical Resolved View vendor source →

Flightcontrol experienced a critical incident on June 15, 2026 affecting Management Plane - ORD and Dashboard and 1 more component, lasting 1h 10m. The incident has been resolved; the full update timeline is below.

Started
Jun 15, 2026, 03:30 PM UTC
Resolved
Jun 15, 2026, 04:41 PM UTC
Duration
1h 10m
Detected by Pingoru
Jun 15, 2026, 03:30 PM UTC

Affected components

Management Plane - ORDDashboardManagement Plane - IADMachines APIManagement Plane - FRAManagement Plane - GRUManagement Plane - LAXDeploymentsManagement Plane - SYDManagement Plane - AMS

Update timeline

  1. investigating Jun 15, 2026, 03:03 PM UTC

    We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

  2. identified Jun 15, 2026, 03:13 PM UTC

    We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing. Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running. We will provide another update within 15 minutes

  3. identified Jun 15, 2026, 03:30 PM UTC

    We have identified the cause of the issue and are working on deploying a fix. Impacted features remain unavailable or degraded at this time. Already running customer applications/machines remain available. MPG clusters remain generally reachable and healthy, however new clusters cannot be provisioned and failovers may not complete. We will provide another update within 15 minutes.

  4. identified Jun 15, 2026, 03:41 PM UTC

    An initial fix has been deployed and we are starting to see platform features recover. Users may still see degraded performance and intermittent failures at this time. We are continuing to address the issue to ensure a full stable recovery.

  5. identified Jun 15, 2026, 03:57 PM UTC

    We continue to seeing degraded performance and increased errors with the Machines API and other platform features at this time. We are continuing to work on fully restoring service.

  6. identified Jun 15, 2026, 04:00 PM UTC

    We are seeing elevated cluster errors with Managed Postgres clusters as the MPG control plane recovers from the API outage. MPG Users may see elevated rates of failing or slow connections, as well as increased primary/replica failovers. The managed postgres team is addressing any degraded clusters. We will provide a further update within 15m.

  7. identified Jun 15, 2026, 04:15 PM UTC

    We have deployed another change and are seeing wider improvements in platform stability across all regions. Performance is trending to normal, though users may still see some degradation at this time. We are continuing to closely monitor to ensure full, stable recovery. We will provide another update in 15m.

  8. monitoring Jun 15, 2026, 04:15 PM UTC

    A fix has been implemented and we are monitoring the results.

  9. resolved Jun 15, 2026, 04:41 PM UTC

    This incident has been resolved and we are seeing all platform functions operate normally.