Kuali had 28 outages in the last 2 years totaling 46h 1m of downtime — averaging 1.2 incidents per month.
There were 28 Kuali outages since September 1, 2024 totaling 46h 1m of downtime. Each is summarised below — incident details, duration, and resolution information.
We will be undergoing database maintenance during our maintenance window Sunday 4/5/26 from 6-10am MDT. Please be aware that Kuali Platform products and APIs will be unavailable for use during this time.
Maintenance will begin as scheduled at 6:00am MDT and finished at 6:15am MDT. The scheduled maintenance has been completed.
Users experienced a brief period where the Platform/Build was not loading due to a temporary disruption in service connectivity.
This issue was caused by a temporary disruption during recent updates that impacted connectivity between services. Service was restored shortly after the issue was identified. We are taking steps to improve deployment processes and reduce the risk of similar disruptions in the future.
Our Canada environments briefly went down due to a temporary issue with our identity service.
Our engineers were unable to identify a specific root cause. However, the issue has been resolved after restarting the Identity service and the Canda instance. We will continue to monitor the system to ensure stability.
We've resolved the issue and apps are loading as expected. We apologize for the inconvenience and are taking steps to prevent similar issues in the future.
We are currently investigating this issue.
We are seeing issues loading Kuali Build and Kuali Next-Gen products in the US region. Our engineers are currently investigating and we will keep you posted on our progress. We apologize for the inconvenience this may be causing.
Our engineers have identified an issue with our service provider, Fly.io, and they are actively working towards a resolution. We will provide additional updates as we have more information.
We are continuing to work on a fix for this issue. While progress has been made, we are still experiencing problems in one of the regions. Our engineers are currently investigating and will continue to provide updates as more information becomes available. We apologize for the inconvenience this may be causing and appreciate your patience.
The environment is back up and operational. Our service provider, Fly.io, has pushed out a fix, and we are continuing to monitor the situation closely to ensure stability. We will share additional updates if anything changes. Thank you for your patience while this was resolved.
While the environments are back up and operational, our service provider, Fly.io, is still working to ensure stability on their end. We are continuing to monitor until the issue is fully resolved.
We are continuing to monitor the status with our provider Fly.io to ensure stability.
The service disruption that occurred today was due to an outage with one of our infrastructure providers, Fly.io. This impacted our ability to deploy services and resulted in temporary unavailability of our environments. The issue has been resolved, and all services are now operational and performing as expected. We are reviewing our deployment processes to further reduce the risk and impact of similar provider-related incidents in the future. Additional details are available on the Fly.io status page at https://status.flyio.net/
We have received reports of workflows that show variables missing on various steps, and integrations displaying errors within forms. We are actively investigating and we will keep you posted.
We've resolved the issue causing the missing variables and errors within integrations, and workflows should be functioning normally again at this time. We are continuing to monitor and we apologize for the inconvenience.
Our team has resolved the issues causing the missing variables within workflows and errors with integrations on forms. We identified an error within a recent deploy that has been corrected. We’ll be looking into ways to avoid this type of issue in the future and we apologize for the inconvenience this caused. If you have any further questions please feel free to reach out to our support team.
Some users may have experienced intermittent errors when accessing pages due to a server entering an unhealthy state. Once identified, our engineers eliminated the unhealthy node and brought a healthy server online to stabilize the environment. During the transition, some customers may have noticed temporary slowness, particularly with actions such as PDF exports or workflow executions. The migration was completed successfully, and the system has since returned to normal operation.
We have received reports of document submissions going to error. We are actively investigating and we will keep you posted.
We've resolved the errors on document submit and the system is fully operational again. We are continuing to monitor and we apologize for the inconvenience.
Our monitoring is complete, and the system is operating normally. The issue was related to networking problems with our service provider, Fly.io. We apologize for the inconvenience and are taking steps to prevent similar issues in the future.
After investigation we found this incident was caused by networking issues with one of our service providers \(Fly.io\). We are looking into adding additional alerts when our system jobs get into a bad state and long term infrastructure improvements that are more robust to help avoid in the future. We apologize for the inconvenience caused by this issue.
We are seeing slowness loading Kuali Build and Kuali Next-Gen products. Our engineers are currently investigating and we will keep you posted on our progress. We apologize for the inconvenience this may be causing.
Our engineers have address the slowness issue with Kuali Build and Kuali Next-Gen products. We’re continuing to monitor and investigate the root cause and we’ll keep you posted as we know more.
The issue affecting Kuali Build and Kuali Next-Gen products is now resolved and our monitoring is complete. Our engineers have deployed a fix to improve the query performance that was causing this slowness. and we have scaled up our systems to ensure improved performance across environments. We apologize for the inconvenience this caused. If you have any further questions or issues please feel free to submit a support ticket.
We discovered long running queries in our codebase that was causing our database to get into a bad state. We’ve improved the query performance by adding additional indexes so results are returned much more efficiently. We’ve also streamlined our process when adding new queries to maintain performance in the future. Lastly, we’re also looking into long term infrastructure improvements that can better distribute performance spikes. We apologize for the inconvenience and we appreciate your patience while we resolved these issues.
We are seeing issues loading Kuali Build and Kuali Next-Gen products in the US region. Our engineers are currently investigating and we will keep you posted on our progress. We apologize for the inconvenience this may be causing.
Build and next generation products should be back up again in US regions. We're continuing to monitor and investigate the root cause.
Environments have been stable and should be fully operational. We had an issue with our database instances getting into a bad state and we're following up with our database service provider (Mongo) to find out what caused. We'll be doing a full retro and provide the post mortem as soon as our investigation is complete. We apologize for any disruption this may have caused.
We discovered long running queries in our codebase that was causing our database to get into a bad state. We’ve improved the query performance by adding additional indexes so results are returned much more efficiently. We’ve also streamlined our process when adding new queries to maintain performance in the future. Lastly, we’re also looking into long term infrastructure improvements that can better distribute performance spikes. We apologize for the inconvenience and we appreciate your patience while we resolved these issues.
We are receiving reports of intermittent issues loading Kuali Build and Kuali Next-Gen products. Our engineers are currently investigating and we will keep you posted on our progress. We apologize for the inconvenience this may be causing.
Kuali Build and Kuali Next-Gen products should now be loading and our engineers are continuing to monitor performance. We'll keep you posted as we have more information.
We wanted to provide an update on the recent errors in the Build/Platform system. These were caused by two separate issues that happened concurrently: 1. An outage with our third-party provider, Fly.io (more info can be found at https://status.flyio.net) 2. A database issue that reduced our system’s capacity We restarted our fly.io instances to restore operations and cycled our Mongo database clusters to alleviate the database issue. The system is now fully operational, and the database is stable. We’re continuing to work with our providers to fully understand what led to the outages and are continue to monitor performance closely to ensure everything runs smoothly. We apologize for the inconvenience and we appreciate your patience while we resolved these issues.
We discovered long running queries in our codebase that was causing our database to get into a bad state. We’ve improved the query performance by adding additional indexes so results are returned much more efficiently. We’ve also streamlined our process when adding new queries to maintain performance in the future. Lastly, we’re also looking into long term infrastructure improvements that can better distribute performance spikes. We apologize for the inconvenience and we appreciate your patience while we resolved these issues.
Users experienced issues loading the application. The problem was due to a database-related issue and was resolved after services recovered.
The issue was caused by increased load on the database, which temporarily impacted application availability. We’re making performance improvements to help prevent future incidents.
Our Canada environments briefly went down due to a temporary issue with one of our service providers, Fly.io.
The root cause was a third-party outage at [Fly.io](http://Fly.io). We’re working with their team to stay ahead of such events and are reviewing our redundancy strategies to minimize future impact.
We experienced a brief disruption in our Canada environments due to a temporary spike in MongoDB traffic. The issue resolved quickly on its own.
The spike in MongoDB traffic temporarily overwhelmed resources, causing a short interruption. We are optimizing database traffic handling and refining monitoring thresholds to help prevent similar issues.
Between 8:52am and 9:10am CT, some users may have experienced intermittent access issues in the production environment. We suspected an issue with Fly.io and noted that they had implemented a temporary fix during the disruption. As minor issues persisted, we rotated infrastructure by scaling up new machines and retiring older ones. Access returned to normal shortly after.
The brief outage was caused by an infrastructure issue at [Fly.io](http://Fly.io), which was mitigated by their temporary fix. We’re tracking improvements on their end and enhancing visibility into upstream provider status.
Users are receiving an error message when initiating forms through links. Initiated forms are available in the user’s Drafts list. Our engineers are investigating the issue and will keep you posted as we have more information.
Our team identified the root cause and have deployed a fix to restore access. We apologize for the inconvenience this caused and if you have any further questions or issues please feel free to submit a support ticket.
To follow up on this issue, our team identified the root cause as an issue retiring form redirects put in place related to the redesign of the popup modals for action list and my documents. This inadvertently affected form initiation on a specific path that hadn’t been caught in our verification. Once identified, we rolled out a fix to restore form creation and we’ll be adding additional testing in the future to avoid this type of issue in the future. Our apologies for the disruption and if you have any further questions please let us know.
We experienced a brief system disruption that quickly self-resolved.
The issue was caused by an unexpected database node outage and quickly self-resolved.
Our mail provider Mailgun.com is currently experiencing degraded performance, so Kuali system notifications are not currently sending as expected and you may see errors on Notification steps in workflow. They are investigating the issue, and we will keep you posted as we have more information.
Our mail provider (Mailgun.com) has confirmed that a fix has been implemented and Kuali system notifications should be sending as expected now. For those Notification workflow steps that went to error you can use the ‘retry’ option to successfully send. Our engineers are continuing to monitor and we apologize for the inconvenience.
This incident has been resolved.
To follow-up on this issue, the issue was caused by degraded performance with our mail provider \(Mailgun.com\). Our monitoring shows that all impacted notifications sent as expected after the fix was implemented. We apologize for the inconvenience and if you have any further questions please let us know.
We are currently investigating an issue where the Build/Platform is not loading or displaying an error message. This issue appears to be affecting customers located in Canada only. Our team is actively working to identify the root cause and restore full functionality as quickly as possible. We apologize for the inconvenience and we will be provide updates as we learn more.
We are continuing to investigate this issue.
We identified that the access issues impacting users in Canada were caused by an overload of database connections. Our engineering team promptly scaled up resources to reduce the load and successfully restore normal performance. We are continuing to monitor the system closely to ensure ongoing stability.
The issue has been resolved. It was caused by an overload of database connections, and we are looking into implementing connection limits and alerts to help prevent future occurrences.
We experienced intermittent connection issues between 3:06 AM and 5:12 AM MT.
Our engineer noted that the proxy restart failed, which caused errors. The manual restart was successful and the issue is now resolved.
We experienced a brief disruption that may have impacted access to some services for a short period.
Our engineers confirmed that one of the database nodes temporarily went down but immediately recovered on its own, restoring service without intervention.
Users are seeing errors or slowness when accessing Build/Platform environments.
The issue was caused by a recent code deployment. Our team identified the problem and rolled back the change, successfully resolving the errors.
We are experiencing unexpected database connection errors that may result in intermittent API and application errors. Our team is actively investigating the issue and working to implement solutions to enhance system stability.
The issue was caused by an unexpected database connection error, which the system automatically resolved. While mitigation steps are in place, these errors still occur sporadically. We are actively working on further improvements to enhance database stability and reduce recurrence. Thank you for your understanding.
We are seeing an issue when loading Build environments, and we are actively troubleshooting the issue.
Our team identified the root cause as an issue with a recently deployed code. Once identified, the code was rolled back to restore service.
We have received reports of intermittent errors with API and submit actions. This may be due to an issue our service provider (fly.io) is actively investigating. Our engineers are monitoring the issue and we will keep you posted with any updates.
We are continuing to investigate this issue.
Our service provider (fly.io) has confirmed several fixes and our monitoring shows improved performance across environments. We are continuing to monitor and we will keep you posted with any updates.
Our monitoring is complete, and our service provider (fly.io) has scaled up their systems and applied fixes to address the issue. The problem is now resolved. We apologize for any inconvenience this may have caused.