Amazee incident

Volumes in read-only mode

Minor Resolved View vendor source →

Amazee experienced a minor incident on February 13, 2025 affecting jp1.lagoon and de3.lagoon and 1 more component, lasting 4d 20h. The incident has been resolved; the full update timeline is below.

Started
Feb 13, 2025, 10:27 AM UTC
Resolved
Feb 18, 2025, 07:07 AM UTC
Duration
4d 20h
Detected by Pingoru
Feb 13, 2025, 10:27 AM UTC

Affected components

jp1.lagoonde3.lagoonau2.lagoonus2.lagoonuk3.lagoon

Update timeline

  1. investigating Feb 13, 2025, 10:27 AM UTC

    We are investigating multiple cases of volumes that were only mounted in read-only mode leading to failed writes for applications. While we are working on a permanent solution, please do reach out to our support should you see your application being impacted by this.

  2. investigating Feb 13, 2025, 11:34 AM UTC

    As a temporary workaround we are immediately reverting some components to a previous version. The impact of this for workloads is like a regular maintenance.

  3. monitoring Feb 13, 2025, 09:37 PM UTC

    Since the rollback we have no longer seen volumes getting falsely mounted as read-only. We will keep monitoring the situation.

  4. resolved Feb 18, 2025, 07:07 AM UTC

    Since we rolled back some components, we have no longer seen issues with volumes getting mounted read-only.