Is Georgia Tech IT down?

Canvas Instructure's multi-factor authentication (2FA) service is currently experiencing an outage, resulting in 504 Gateway Timeout errors for users attempting to authenticate. This issue is impacting access to Canvas services that require 2FA verification. The service disruption is under investigation, and teams are working to identify the root cause and restore normal functionality as quickly as possible.

"Cloudfront has a global outage. We investigating the issue together with AWS experts. You can follow updates from their team here: https://health.aws.amazon.com/health/status"

A fix has been implemented, and Canvas is now accessible.

Following a power outage and a firewall upgrade change implementation, a substantial number of telecommunications nodes and network switches became unreachable, resulting in service degradation across affected areas.

The Georgia Tech team identified a high-priority dataset located within the cedar_cos volume that is being retrieved at very slow rates (reported at approximately 5–10 MB/s), delaying access to important research and operational data. Our vendor has been actively investigating the issue, reviewing audit logs, analyzing S3 GET activity, and exploring potential remediation options. They have determined that the requested data cannot easily be targeted at the individual directory level because the underlying StorageGRID object mapping is performed at the volume level rather than the client-share folder level. Our vendor is considering a temporary maintenance strategy to reduce non-essential retrieval activity and prioritize access to the critical dataset while longer-term engineering fixes continue to be evaluated. A meeting is scheduled between the vendor and Georgia Tech stakeholders to discuss mitigation options and next steps.

**What’s happening?** CEDAR storage has been made unavailable starting today, July 9, at 3.20pm ET. At this time, there is not an estimated day/time for when services will be returned. Emergency work is necessary to mitigate slow data retrieval times. **When is it happening?** CEDAR storage became inaccessible starting today, July 9, at 3.20pm ET. **Why is it happening? ** The Georgia Tech team identified a high-priority dataset located within the cedar_cos volume that is being retrieved at very slow rates (reported at approximately 5–10 MB/s), delaying access to important research and operational data. Our vendor has been actively investigating the issue, reviewing audit logs, analyzing S3 GET activity, and exploring potential remediation options. The vendor determined that the requested data cannot easily be targeted at the individual directory level because the underlying StorageGRID object mapping is performed at the volume level rather than the client-share folder level. Our vendor has proposed a temporary maintenance strategy to reduce non-essential retrieval activity and prioritize access to the critical dataset while longer-term engineering fixes continue to be evaluated. **Who is affected?** All CEDAR users. **What do you need to do?** Access and any data connections to/from CEDAR was automatically interrupted and cancelled at 3.20pm. At this time, users do not need to take action. **Who should you contact for questions?** For any questions, please contact [email protected]. More information will be provided as it is available.

Troubleshooting work with storage vendor continues. Core engineering resources are being engaged.

Troubleshooting will continue tomorrow morning. Vendor is analyzing logs and will provide additional updates as we get them.

We finally settled on the best performance parameters to transfer a critical file set from CEDAR. NetApp engineering continues root cause analysis and are working on a patch to fix the performance issues. Storage reamins closed to all users while the critical transfer completes. Issue has been escalated to the executive VP for support and engineering.

Critical file sets are being transferred, while NetApp continues working on permanent fixes that will allow us to open the file system with the expected transfer rates. GT directors will meet with the NetApp VP of support and engineering in the next couple of hours. Another status call with NetApp is scheduled for 4pm.

NetApp engineering updated an executable on one of the storage nodes. Results are promising and more analysis is needed. In the meantime, the critical data transfer continues.

CEDAR file system is open to all users. The patched code remains in place and all servers are showing the same level of performance the system had before upgrading to 11.8 code. We'll continue watching the status at least until tomorrow at noon.

OIT Network Engineering is aware of issues affecting the ability of users on the campus network to access resources/services hosted by GTRI. This also includes the ability of users to connect to the GTRI-Device wireless network in non-GTRI spaces. Engineers are actively investigating, and further information will be shared as it becomes available.

The issues affecting connectivity between Campus and GTRI, including the functionality of GTRI-Device in non-GTRI spaces, have been identified and a fix put in place. Users should now be able to access resources/services hosted by GTRI from the campus network. Network Engineering will continue to monitor for any recurrence of the issue.

Mycloud logins are currently failing- users may get an "incorrect username and password" error even when entering the correct username and password. Support is currently looking into the issue