Amazee incident

Delayed log processing

Major Resolved View vendor source →

Amazee experienced a major incident on July 16, 2025 affecting Lagoon API and Lagoon Logs (Kibana), lasting 17h 56m. The incident has been resolved; the full update timeline is below.

Started
Jul 16, 2025, 02:55 PM UTC
Resolved
Jul 17, 2025, 08:52 AM UTC
Duration
17h 56m
Detected by Pingoru
Jul 16, 2025, 02:55 PM UTC

Affected components

Lagoon APILagoon Logs (Kibana)

Update timeline

  1. investigating Jul 16, 2025, 02:55 PM UTC

    There is a delay in log processing and recent logs are not available in the logging system. Real time logs can be access through the Lagoon CLI: https://docs.amazee.io/cloud/logging/#real-time-container-logs-via-lagoon-cli

  2. identified Jul 16, 2025, 04:02 PM UTC

    We are rolling out a potential solution which can cause connectivity issues with the Lagoon API.

  3. monitoring Jul 16, 2025, 04:14 PM UTC

    While the logs are being processed the logging system can be unavailable at times.

  4. monitoring Jul 17, 2025, 04:59 AM UTC

    The processing of some older logs is still ongoing.

  5. resolved Jul 17, 2025, 08:52 AM UTC

    The historical logs have been processed and are available in the logging stack.