NERC Outage History

Type: Maintenance Duration: 3 hours Affected Components: NERC RED HAT OPENSHIFT AI (RHOAI), OPENSHIFT CONTROLLER, NERC OPENSHIFT WEB CONSOLE, OPENSHIFT WORKERS NODES Apr 15, 13:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time to add remaining H100s with RDMA support. Apr 15, 13:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 15, 16:00:00 GMT+0 - Completed - Maintenance has completed successfully

Type: Incident Duration: 1 day and 17 hours Mar 20, 18:27:22 GMT+0 - Identified - We are continuing to work on a fix for this incident. Mar 19, 04:00:00 GMT+0 - Investigating - We have identified an issue with the "**Cumulative Daily Charges for Month**" value shown in daily cost notifications for OpenStack allocations. Since the March ColdFront maintenance on **03/19**, some OpenStack storage charges have not been fully included in this field, causing reported cumulative charges to be lower than actual costs. As a result, some users may have already exceeded their cost alert thresholds without being notified. Until this is fixed, please do not rely solely on the “**Cumulative Daily Charges for Month**” value and instead **review** your approved **storage allocation quota** on the NERC's ColdFront. Mar 20, 21:00:28 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 35 minutes Affected Components: OPENSTACK EC2 API PROXY (EC2API), , OPENSTACK DASHBOARD (HORIZON), NERC COLDFRONT, NERC OPENSHIFT WEB CONSOLE, , NERC OPENSTACK → NERC OPENSHIFT CONTAINER PLATFORM - PRODUCTION → Mar 5, 17:36:47 GMT+0 - Resolved - This incident has been resolved. Mar 5, 15:02:09 GMT+0 - Investigating - We are currently investigating this incident. Mar 5, 16:47:15 GMT+0 - Investigating - We are currently investigating this incident. This is an isolated issue that is affecting Harvard users only! Mar 5, 17:18:07 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result.

Type: Incident Duration: 3 days, 19 hours and 25 minutes Affected Components: OPENSTACK BLOCK STORAGE (CINDER), STORAGE, OPENSTACK OBJECT STORE (SWIFT) Jan 9, 20:36:55 GMT+0 - Investigating - We are currently investigating this incident. Jan 12, 14:17:21 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have informed users who are hosting large amount of fragmented objects and begin cleaning up some. Jan 13, 16:01:52 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 5 minutes Affected Components: NERC WEBSITE Dec 11, 15:20:31 GMT+0 - Investigating - OTHER WEBSITES & TOOLS - NERC WEBSITE cannot be accessed at the moment. This incident was created by an automated monitoring service. Dec 11, 15:25:57 GMT+0 - Resolved - OTHER WEBSITES & TOOLS - NERC WEBSITE is now operational! This update was created by an automated monitoring service.