Scalyr incident

ingestions and queries are affected on app.scalyr.com, app.eu.scalyr.com, app.dataset.com, and app.eu.dataset.com

Notice Resolved View vendor source →

Scalyr experienced a notice incident on January 16, 2025, lasting 5h 42m. The incident has been resolved; the full update timeline is below.

Started
Jan 16, 2025, 06:39 PM UTC
Resolved
Jan 17, 2025, 12:22 AM UTC
Duration
5h 42m
Detected by Pingoru
Jan 16, 2025, 06:39 PM UTC

Update timeline

  1. investigating Jan 16, 2025, 06:39 PM UTC

    We are currently investigating this issue.

  2. investigating Jan 16, 2025, 07:10 PM UTC

    We are in the process of rolling back the change and restarting the affected servers. Some accounts may observe a gradual recovery of their logs.

  3. identified Jan 16, 2025, 07:48 PM UTC

    A misconfiguration deployment caused server resource allocation issues. A rollback and server restart have been initiated to address the problem.

  4. monitoring Jan 16, 2025, 10:01 PM UTC

    Recovery is currently in progress. The estimated time for full ingestion restoration is approximately 60 to 90 minutes from now.

  5. resolved Jan 17, 2025, 12:22 AM UTC

    The backlog data caused by the incident has been fully recovered in all regions