University of Alaska experienced a critical incident on June 30, 2025 affecting Banner 9, lasting 19h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 30, 2025, 08:04 PM UTC
The OIT EAS team is aware of a problem stopping all Production Banner Printing from working correctly. EAS believes they have identified the issue, and are working with the vendor to resolve it. Thank you for your patience, we will try to have an update every thirty minutes.
- identified Jun 30, 2025, 08:38 PM UTC
The OIT Enterprise Applications Services team is still working to resolve this issue. Thank you for your continued patience.
- identified Jun 30, 2025, 09:08 PM UTC
The OIT Enterprise Applications Services team is still working with the vendor to resolve this issue. OIT EAS has implemented a manual workaround and users should start seeing their expected printed documents within the next several minutes, if they're not already there. Thank you for your continued patience.
- identified Jun 30, 2025, 09:35 PM UTC
The OIT Enterprise Applications Services team is continuing to work with the vendor to resolve this issue. OIT EAS has implemented a manual workaround and users should start seeing their expected printed documents within the next several minutes, if they're not already there. Thank you for your continued patience.
- identified Jun 30, 2025, 10:04 PM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident!
- identified Jun 30, 2025, 10:34 PM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.
- identified Jun 30, 2025, 11:04 PM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.
- identified Jun 30, 2025, 11:36 PM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.
- identified Jul 01, 2025, 12:05 AM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Thank you again for your patience as we work to resolve this incident.
- identified Jul 01, 2025, 01:01 AM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Print jobs will continue to be manually pushed throughout the night. Thank you again for your patience as we work to resolve this incident.
- identified Jul 01, 2025, 02:02 AM UTC
The OIT EAS team is continuing to work with the vendor to resolve this issue, and there is no new information available at this time. Banner print jobs should still be processing with the manual workaround implemented by EAS. Print jobs will continue to be manually pushed throughout the night. Thank you again for your patience as we work to resolve this incident.
- identified Jul 01, 2025, 03:07 AM UTC
There is no new information available at this time, however Banner print jobs should still be processing with the manual workaround implemented by EAS, and print jobs will continue to be manually pushed throughout the night. This will be the last update posted tonight, and we will begin updating this status page again tomorrow at 7:30am. Thank you for your continued patience during this incident.
- monitoring Jul 01, 2025, 04:44 AM UTC
A fix has been implemented, and the Banner Printing outage appears to be resolved. We are actively monitoring the issue, but pending prints are now flowing and being processed. Thank you for your patience during this incident.
- resolved Jul 01, 2025, 03:29 PM UTC
The OIT Enterprise Applications Services team has resolved the issue with Banner Printing. After working with the vendor, it was determined there was an issue on the vendor's side, who will be reviewing how the error occurred. We appreciate everyone's patience while this issue was being resolved.
- postmortem Jul 01, 2025, 08:59 PM UTC
# Background Banner printing is the by which receipts, vendor checks, transcripts, and several other documents are printed. When banner printing is not working it holds up several business processes. The printing setup has multiple components that include an on-premise print application, a web end point to query for queued printing, and several configurations in the core product. The web end point, part of Banner 9 General Self Service, has to be configured for guest authentication in order to be successful. # Break Down of the Problem On June 30th, at approximately 9:49 AM the first report of Banner printing not working for a receipt printer came in. The responding technician evaluated and determined the scope was all of Banner printing that relied on the restful endpoint. A priority 1 ticket was put in with the vendor to evaluate the current environment’s configuration for the restful endpoint. A UA technician’s evaluation of local log data for the printer application determined that Banner printing for production was unavailable since June 27 at 10:08pm. During the outage time, a responding technician implemented a work around to the production process not functioning, which was done by manual execution that pulled down the rendered PDF of the print job and sent it to the destination printer. At approximately 8:34pm the vendor working with UA technicians was able to resolve the issue with the web endpoint. # Target State / Goal Banner printing services should be available to customers 24/7 except during pre-defined maintenance windows. # Root Cause Analysis During a recent planned change activity starting on June 27th at 9pm, certificates were being updated across the Ellucian hosted production environment. During this update, the web endpoint was redeployed. After the endpoint returned to service, the process used to retrieve print jobs failed to be able to retrieve them with an HTTP 401 error indicating that the account wasn’t authorized. Technicians validating the change did not notice this as the web location was validated via other means. Technicians escalated to the vendor when first noticed on June 30th at 9:50am, and soon after shared with the vendor their expectation that a setting for guest authentication wasn’t correctly configured. Vendor believed they verified the setting was in place, and continued to evaluate other items. Vendor determined at 8pm that the setup parameter was incorrect and changed around April 10, for an undetermined reason. The parameter was corrected to match the same configuration in the TEST environment and the web end point redeployed restoring service. # Develop Countermeasures * Engage vendor to prevent future similar failures for their deployments. * Require technicians to validate that after a Banner 9 General Self Service Deployment that the Banner Print Application can connect to the web endpoint. * Review ability to set up a monitor for immediately alerting around failures for the Banner Print Application to connect to the Banner 9 General Self Service endpoint. # Implementation of Countermeasures * _Jul 1 11:00 AM: Engage vendor about how they will be able to prevent this from occurring in the future._ * July 16: Create documentation for validators to follow to ensure the Banner 9 General Self Service deployments are correctly accepting connections from the Banner Print Application. * July 16: Complete review of ability to set up a monitor for immediately alerting technicians at the first stage of failure for the Banner Print application to connect to the Banner 9 GeneralSSB endpoint. # Follow Up / Review * July 2: Ensure vendor is working through an internal procedure to ensure this issue is prevented in the future. * July 18: Verify documentation of validation was created for post validation of Banner 9 General SSB * July 18: Review/Discuss feasibility of monitoring the connection status for immediately alerting. If feasible, define follow up tasks for improvement.