Georgia Tech IT incident

CEDAR – Temporarily Pause Adding/Writing Data

Minor Resolved View vendor source →

Georgia Tech IT experienced a minor incident on September 25, 2026 affecting PACE and PACE (CEDAR Storage), lasting 4d 23h. The incident has been resolved; the full update timeline is below.

Started
Sep 25, 2026, 04:49 PM UTC
Resolved
Sep 30, 2026, 03:57 PM UTC
Duration
4d 23h
Detected by Pingoru
Sep 25, 2026, 04:49 PM UTC

Affected components

PACEPACE (CEDAR Storage)

Update timeline

  1. monitoring Sep 25, 2026, 04:49 PM UTC

    Dear CEDAR users, The CEDAR storage system is approaching capacity. The addition of 2.4 PB of new storage is expected by mid-October. When possible, please temporarily stop writing and modifying data to CEDAR while mitigations are implemented. Users should not delete data – pausing writing and modifying data while work is carried out with the vendor is sufficient. Read-only operations are safe to continue. Thank you for helping us keep CEDAR’s performance optimal. Please contact us at [email protected] with questions. We will follow up with additional details on CEDAR’s expansion once they are available. Thank you, The PACE Team

  2. monitoring Sep 25, 2026, 05:17 PM UTC

    We are closely monitoring storage capacity for the CEDAR service, which is approaching its available limits. A capacity expansion is planned for mid-October, but usage is expected to continue increasing before that work is completed. At this time, a small number of users have reported delays when accessing or updating data. Based on current performance trends, additional users may begin to experience similar issues. OIT is working with the vendor to identify options for recovering capacity and reducing the risk of service degradation. We are also requesting that users avoid creating, modifying, or transferring large amounts of data unless necessary to help preserve service stability. Users should not delete data at this time. Impact: Users may experience slower-than-normal performance when accessing, updating, or storing data in CEDAR."

  3. monitoring Sep 25, 2026, 08:29 PM UTC

    PACE and the CEDAR vendor have identified and begun implementing measures to create additional capacity on CEDAR. More capacity is expected to become available Sunday evening as automated cleanup processes continue to run. We have verified that storage space is beginning to be reclaimed and will continue to closely monitor capacity utilization and system performance as these cleanup activities progress. At this time, these actions are expected to reduce pressure on the system and improve performance for affected users. As a precaution, users should continue to avoid creating, modifying, or transferring large amounts of data unless necessary. Read-only operations remain safe to perform. Additional updates will be provided as more information becomes available

  4. monitoring Sep 28, 2026, 04:58 PM UTC

    Deletions over the weekend reduced CEDAR's capacity from 93% to 90% and traffic rates are currently low. We continue to ask users to not add data to CEDAR for now, and will have more clear recommendations after meeting with the vendor this afternoon

  5. monitoring Sep 28, 2026, 08:56 PM UTC

    PACE is continuing to expunge snapshots to make space available, and is looking into accelerating the schedule for bringing the new capacity online with the goal of 2 additional PB being available by end of week. At this time, our recommendations for users are still the same.

  6. monitoring Sep 29, 2026, 07:40 PM UTC

    CEDAR capacity utilization has decreased to 88% following successful snapshot cleanup efforts and reduced intake of new data. PACE is continuing to engage the CEDAR vendor to increase capacity. Users are asked to continue to limit the creation, modification, and transfer of large amounts of data whenever possible.