Macaroon Auth + Machines API Issues

Summary

Macaroon Auth + Machines API Issues

This incident has been resolved and we are seeing all platform functions operate normally.

• Started

  Jun 15, 2026 · 3:03 PM

• Resolved

  Jun 15, 2026 · 4:41 PM

• Duration

  1h 37m

• Severity

  Critical

• ◐

  Investigating

  Jun 15 · 3:03 PM Fly.io

  We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

• ◆

  Identified

  Jun 15 · 3:13 PM Fly.io

  We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing. Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running. We will provide another update within 15 minutes

• ◆

  Identified

  Jun 15 · 3:30 PM Fly.io

  We have identified the cause of the issue and are working on deploying a fix. Impacted features remain unavailable or degraded at this time. Already running customer applications/machines remain available. MPG clusters remain generally reachable and healthy, however new clusters cannot be provisioned and failovers may not complete. We will provide another update within 15 minutes.

• ◆

  Identified

  Jun 15 · 3:41 PM Fly.io

  An initial fix has been deployed and we are starting to see platform features recover. Users may still see degraded performance and intermittent failures at this time. We are continuing to address the issue to ensure a full stable recovery.

• ◆

  Identified

  Jun 15 · 3:57 PM Fly.io

  We continue to seeing degraded performance and increased errors with the Machines API and other platform features at this time. We are continuing to work on fully restoring service.

• ◆

  Identified

  Jun 15 · 4:00 PM Fly.io

  We are seeing elevated cluster errors with Managed Postgres clusters as the MPG control plane recovers from the API outage. MPG Users may see elevated rates of failing or slow connections, as well as increased primary/replica failovers. The managed postgres team is addressing any degraded clusters. We will provide a further update within 15m.

• ◆

  Identified

  Jun 15 · 4:15 PM Fly.io

  We have deployed another change and are seeing wider improvements in platform stability across all regions. Performance is trending to normal, though users may still see some degradation at this time. We are continuing to closely monitor to ensure full, stable recovery. We will provide another update in 15m.

• ◉

  Monitoring

  Jun 15 · 4:15 PM Fly.io

  A fix has been implemented and we are monitoring the results.

• ✓

  Resolved

  Jun 15 · 4:41 PM Fly.io

  This incident has been resolved and we are seeing all platform functions operate normally.