Civitas Learning experienced a minor incident on April 24, 2025 affecting Degree Map and Data Warehouse (Explore) and 1 more component, lasting 4d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 24, 2025, 01:25 PM UTC
The issue has been identified and a fix is being developed.
- monitoring Apr 24, 2025, 07:47 PM UTC
A fix has been implemented and we are monitoring the results.
- identified Apr 24, 2025, 09:03 PM UTC
Some additional issues have been identified and we are working to resolve as quickly as possible
- identified Apr 25, 2025, 02:20 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Apr 25, 2025, 02:21 PM UTC
A fix has been implemented and we are monitoring the results. Data freshness is starting to come back within the 24-48 hour threshold.
- resolved Apr 28, 2025, 04:22 PM UTC
On Tuesday April 22nd we began to notice our monitors indicating a number of workflow failures. Initially our logs pointed to DNS failures with our cloud provider. Upon troubleshooting the root cause, we identified an unhealthy node in one of our clusters that was still accepting traffic but unable to properly process data since it was in a failure state. Remediation was to manually remove the unhealthy node from the cluster at which point the issue was resolved.