Blackboard incident
Illuminate Data Refresh Issue in US & EU regions
Blackboard experienced a major incident on July 8, 2025 affecting Illuminate, lasting 6d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 08, 2025, 03:40 PM UTC
We are currently experiencing issues with the data pipeline in US and EU regions, which may result in delays in the refresh of Blackboard data for clients in these regions. Our team is actively working to resolve the issue and restore full functionality.
- investigating Jul 08, 2025, 10:37 PM UTC
Our engineering team continues to investigate the issue to identify the root cause.
- investigating Jul 09, 2025, 06:10 PM UTC
TLS handshake errors in the connection between Blackboard and the Snowflake database have been identified as a potential root cause. We are actively investigating the vendor’s proposed mitigations.
- identified Jul 09, 2025, 11:22 PM UTC
We have identified the root cause and the engineering team is working on a fix, with the goal of including it in the upcoming Blackboard 3900.121.0 and 3900.123.0 production releases.
- identified Jul 10, 2025, 06:34 PM UTC
It has been determined that the underlying issue was the result of a vendor change included in the recent Blackboard release that interrupted communication between Blackboard and Illuminate for some requests involving large amounts of data. The fix will be applied in conjunction with the next Blackboard update. For production environments, this means the fix will be deployed with the 3900.121.0 update today. For test and staging environments, the fix will be deployed on Tuesday with the 3900.123.0 update.
- monitoring Jul 14, 2025, 04:25 PM UTC
The fix is now live in production for all instances and has also been applied to affected instances in lower environments (test and staging). We are monitoring the results.
- resolved Jul 14, 2025, 05:43 PM UTC
This incident has been resolved.