Georgia Tech IT experienced a minor incident on July 9, 2026 affecting Academic Services and Academic Services (PACE), lasting 3d 20h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 09, 2026, 05:09 PM UTC
The Georgia Tech team identified a high-priority dataset located within the cedar_cos volume that is being retrieved at very slow rates (reported at approximately 5–10 MB/s), delaying access to important research and operational data. Our vendor has been actively investigating the issue, reviewing audit logs, analyzing S3 GET activity, and exploring potential remediation options. They have determined that the requested data cannot easily be targeted at the individual directory level because the underlying StorageGRID object mapping is performed at the volume level rather than the client-share folder level. Our vendor is considering a temporary maintenance strategy to reduce non-essential retrieval activity and prioritize access to the critical dataset while longer-term engineering fixes continue to be evaluated. A meeting is scheduled between the vendor and Georgia Tech stakeholders to discuss mitigation options and next steps.
- identified Jul 09, 2026, 07:36 PM UTC
**What’s happening?** CEDAR storage has been made unavailable starting today, July 9, at 3.20pm ET. At this time, there is not an estimated day/time for when services will be returned. Emergency work is necessary to mitigate slow data retrieval times. **When is it happening?** CEDAR storage became inaccessible starting today, July 9, at 3.20pm ET. **Why is it happening? ** The Georgia Tech team identified a high-priority dataset located within the cedar_cos volume that is being retrieved at very slow rates (reported at approximately 5–10 MB/s), delaying access to important research and operational data. Our vendor has been actively investigating the issue, reviewing audit logs, analyzing S3 GET activity, and exploring potential remediation options. The vendor determined that the requested data cannot easily be targeted at the individual directory level because the underlying StorageGRID object mapping is performed at the volume level rather than the client-share folder level. Our vendor has proposed a temporary maintenance strategy to reduce non-essential retrieval activity and prioritize access to the critical dataset while longer-term engineering fixes continue to be evaluated. **Who is affected?** All CEDAR users. **What do you need to do?** Access and any data connections to/from CEDAR was automatically interrupted and cancelled at 3.20pm. At this time, users do not need to take action. **Who should you contact for questions?** For any questions, please contact [email protected]. More information will be provided as it is available.
- identified Jul 09, 2026, 09:33 PM UTC
Troubleshooting work with storage vendor continues. Core engineering resources are being engaged.
- identified Jul 09, 2026, 11:22 PM UTC
Troubleshooting will continue tomorrow morning. Vendor is analyzing logs and will provide additional updates as we get them.
- identified Jul 10, 2026, 03:22 PM UTC
We finally settled on the best performance parameters to transfer a critical file set from CEDAR. NetApp engineering continues root cause analysis and are working on a patch to fix the performance issues. Storage reamins closed to all users while the critical transfer completes. Issue has been escalated to the executive VP for support and engineering.
- identified Jul 10, 2026, 06:31 PM UTC
Critical file sets are being transferred, while NetApp continues working on permanent fixes that will allow us to open the file system with the expected transfer rates. GT directors will meet with the NetApp VP of support and engineering in the next couple of hours. Another status call with NetApp is scheduled for 4pm.
- identified Jul 10, 2026, 09:02 PM UTC
NetApp engineering updated an executable on one of the storage nodes. Results are promising and more analysis is needed. In the meantime, the critical data transfer continues.
- monitoring Jul 12, 2026, 06:05 PM UTC
CEDAR file system is open to all users. The patched code remains in place and all servers are showing the same level of performance the system had before upgrading to 11.8 code. We'll continue watching the status at least until tomorrow at noon.