ByWater Solutions Outage History

Started Oct 21, 2026, 05:54 PM UTC· resolved Oct 21, 2026, 08:14 PM UTC

1. identified Oct 21, 2026, 05:54 PM UTC

   Security updates are being applied to Coral systems, which will be unavailable for a period of time to allow for patching and testing. We will update here again once testing has completed.

2. resolved Oct 21, 2026, 08:14 PM UTC

   Our team has finished the patching and testing of the Coral instances and brought the sites back online. If you notice any bugs with Coral, please open a ticket with our support team.

Read the full incident report →

Started Oct 21, 2026, 01:58 PM UTC· resolved Oct 21, 2026, 06:27 PM UTC

1. identified Oct 21, 2026, 01:58 PM UTC

   Security updates are being applied to Aspen systems, which may result in temporary (less than 5 minutes) interruptions to Aspen services. Not all Aspen partners may notice this interruption. Urgent security updates are applied during operation hours only when the need is decided to be urgent and also critical. We appreciate your patience while we take these actions to ensure your Aspen servers stay secure. We will have a follow up update once these updates have been applied.

2. resolved Oct 21, 2026, 06:27 PM UTC

   Our systems team has completed testing for Aspen Discovery. All partner production and test Aspen instances have been patched and are operational.

Read the full incident report →

Started Oct 16, 2026, 12:56 PM UTC· resolved Oct 19, 2026, 03:53 PM UTC

1. investigating Oct 16, 2026, 12:56 PM UTC

   The systems team have identified increased bot activity over the past few hours. We are taking steps to mitigate interruptions to partner libraries.

2. resolved Oct 19, 2026, 03:53 PM UTC

   This has been mitigated. We will continue to monitor our systems for bot activity.

Read the full incident report →

Started Sep 24, 2026, 02:35 PM UTC· resolved Sep 24, 2026, 06:14 PM UTC

1. investigating Sep 24, 2026, 02:35 PM UTC

   Our team has been notified of an issue with libraries using the Sierra ILS with Aspen Discovery. This issue is with Sierra, and it is recommended to contact III for more information. We will update this as we learn more, but suggest contact with III for regular updates.

2. identified Sep 24, 2026, 03:57 PM UTC

   The issue with Sierra ILS has been confirmed, and to our knowledge their teams are working on a fix. Please direct all issues with Sierra post-ILS upgrade to III as we do not have access to that system.

3. resolved Sep 24, 2026, 06:14 PM UTC

   The issue observed with Sierra ILS libraries using Aspen Discovery has been resolved by III and library staff should be able to log in as expected. The Aspen indexers are working to catch up, and may take some time to fully update due to this interruption. Please let us know if you continue to see login issues, and thank you for your patience as the indexers complete their work.

Read the full incident report →

Started Sep 21, 2026, 02:19 PM UTC· resolved Sep 21, 2026, 03:02 PM UTC

1. investigating Sep 21, 2026, 02:19 PM UTC

   Our operations team has identified an issue with a subset of our hosted Koha partners, which may result in background jobs being backed up or not completing as expected. The team is working on restoring these processes and will have an update in one hour.

2. identified Sep 21, 2026, 02:37 PM UTC

   Our team is currently rolling out a fix for those affected partners, which we will have completed within the next 30 minutes. We will have another update after we have completed our checks.

3. resolved Sep 21, 2026, 03:02 PM UTC

   Our systems team has identified the root cause of the issue to have been scheduled maintenance on the databases, which interrupted the services for long enough that the services were not able to successfully recover automatically. The database updates themselves were not interrupted, and the databases are healthy. We are currently also integrating a more resilient configuration for these services to ensure that this does not happen again. If you continue to see issues with your background jobs after the posting of this update, please open a ticket with us and let us know.

Read the full incident report →

Started Sep 14, 2026, 08:24 PM UTC· resolved Sep 15, 2026, 04:38 PM UTC

1. investigating Sep 14, 2026, 08:24 PM UTC

   Our operations team has identified an issue with Libki Server hosting. They are currently working on a solution, and will have an update shortly

2. investigating Sep 15, 2026, 01:53 PM UTC

   Our team continues to investigate the issue affecting Libki server hosting.

3. resolved Sep 15, 2026, 04:38 PM UTC

   The issue impacting some partner Libki sites' certificates has been resolved. If you have a problem which persists after this update, please open a ticket with us, and our support team will be happy to help!

Read the full incident report →

Started Sep 09, 2026, 04:18 PM UTC· resolved Sep 09, 2026, 07:21 PM UTC

1. investigating Sep 09, 2026, 04:18 PM UTC

   Our operations team has identified an issue with searching within Koha. They're investigating and will provide an update in the next 30 minutes. You may get no results or timed out results.

2. identified Sep 09, 2026, 04:26 PM UTC

   Things appear to be recovering, but the team is still investigating. Will provide another update within the next 30 minutes.

3. monitoring Sep 09, 2026, 04:53 PM UTC

   Systems are maintaining their availability. There was an influx of usage on the elastic cluster (which powers searching). We will be evaluating methods to mitigate slowness so future surges do not impact performance.

4. resolved Sep 09, 2026, 07:21 PM UTC

   Searching is working, and we will perform maintenance after hours to help prevent this from happening again. Important Note: Anything that was being indexed at that time may have not made it to the elastic indexes. A quick fix is to just edit and re-save a given marc record. If this is affecting a lot of records, we can run a process to index your missing records, please submit a ticket if this is the case.

Read the full incident report →

Started Aug 05, 2026, 06:13 PM UTC· resolved Sep 09, 2026, 04:16 PM UTC

1. investigating Aug 05, 2026, 06:13 PM UTC

   We are currently working with our upstream provider to identify whether technical issues currently identified for their services is impacting our hosted Koha. We are investigating with our provider and will give updates as we have them.

2. investigating Aug 05, 2026, 07:28 PM UTC

   Our systems team is continuing to work with our upstream provider to ensure their current outage has minimal impact to our hosted services. Another update will come in one hour.

3. monitoring Aug 05, 2026, 08:19 PM UTC

   Our systems team is continuing to work with our upstream provider and monitor the situation. Further updates will come as we receive new information from Rackspace.

Read the full incident report →

Started Aug 04, 2026, 09:30 PM UTC· resolved Aug 04, 2026, 09:48 PM UTC

1. investigating Aug 04, 2026, 09:30 PM UTC

   Our operations team has identified an issue with Cloudflare. We have identified the issue and are currently working on implementing a fix for this. We will have another update in 30 minutes.

2. resolved Aug 04, 2026, 09:48 PM UTC

   The Cloudflare issue has been fixed, and systems are showing recovery. If you are still experiencing issues accessing your system, please open a ticket with us and let us know!

Read the full incident report →

Started Jul 09, 2026, 05:00 PM UTC· resolved Jul 20, 2026, 01:53 PM UTC

1. investigating Jul 09, 2026, 05:00 PM UTC

   Due to an inter-server dependency on some Discovery systems, branded LiDA apps may not be able to establish a connection with the library. The community LIDA app is also impacted. The operations team is working on deploying a mitigation to alleviate LiDA outages, and broader underlying changes to remove this dependency entirely.

2. identified Jul 09, 2026, 07:08 PM UTC

   Branded LiDA apps are now functional. The community app remains offline

3. investigating Jul 10, 2026, 05:58 PM UTC

   We are continuing to work on restoring access to the Community LiDA app.

4. investigating Jul 13, 2026, 02:25 PM UTC

   Our systems team is continuing to work on the community Aspen LiDA app

5. monitoring Jul 14, 2026, 02:55 PM UTC

   Bugs in Community LiDA have been identified, patched and submitted to the app stores for review.

6. monitoring Jul 14, 2026, 03:00 PM UTC

   The Community LiDA update has been accepted by Google Play store, and is available for download

7. monitoring Jul 14, 2026, 09:11 PM UTC

   Apple has accepted the Community LiDA 26.06.02 update. Some partner libraries will need a code update to their Aspen Discovery, scheduled for tonight's routine maintenance window.

8. resolved Jul 20, 2026, 01:53 PM UTC

   Community Aspen LiDA is once again functional for partner libraries

Read the full incident report →

Started Jun 24, 2026, 04:01 PM UTC· resolved Jul 07, 2026, 05:41 PM UTC

1. investigating Jun 24, 2026, 04:01 PM UTC

   There is a bug in the latest version of LiDA (26.05.00) that causes the app to crash after login. The bug affects all platforms. Our developers are working to resolve the issue as quickly as possible. We will update this page with our progress.

2. resolved Jun 24, 2026, 07:17 PM UTC

   Our developers have submitted a new build to Apple and Google to resolve this issue. Once we have confirmed that the build is successful, we will deploy it to our partners.

3. resolved Jun 25, 2026, 12:52 PM UTC

   A fix has been created and is being distributed through the app stores to all affected partners.

4. resolved Jul 07, 2026, 05:41 PM UTC

   LiDA is operational on Android. We are awaiting review of the iOS build.

Read the full incident report →

Started Apr 23, 2026, 03:00 PM UTC· resolved Apr 23, 2026, 04:09 PM UTC

1. investigating Apr 23, 2026, 03:00 PM UTC

   Our operations team has identified an issue with Koha connections. They are currently investigating and will have an update in 30 minutes.

2. resolved Apr 23, 2026, 04:09 PM UTC

   The operations team has identified the issue, taken action to remediate the issue, and is currently monitoring for any further developments. Any partner still experiencing interruptions should call or open a ticket for us to assist.

Read the full incident report →

Started Apr 22, 2026, 06:19 PM UTC· resolved Apr 22, 2026, 09:00 PM UTC

1. investigating Apr 22, 2026, 06:19 PM UTC

   Our operations team has identified an issue with our hosted Metabase instances. They are currently investigating and will have an update in one hour.

2. resolved Apr 22, 2026, 07:43 PM UTC

   Our team is continuing to work on Metabase. Most instances are now operational. We will have another update in one hour.

3. resolved Apr 22, 2026, 09:00 PM UTC

   All systems are back up and we are continuing to monitor.

Read the full incident report →

Started Mar 18, 2026, 02:20 PM UTC· resolved Mar 18, 2026, 04:12 PM UTC

1. investigating Mar 18, 2026, 02:20 PM UTC

   Our operations team has identified an issue affecting a subset of our partners, resulting in 500 errors when searching Koha. They are currently investigating and will have an update in one hour.

2. investigating Mar 18, 2026, 03:21 PM UTC

   Our DevOps team continues to investigate what caused the 500 errors and analytics errors. We will have another update in one hour.

3. resolved Mar 18, 2026, 04:12 PM UTC

   The issue affecting a subset of partners that caused 500 errors during Koha search requests has been identified as related to Elasticsearch and is now resolved. Our team has taken corrective action to restore normal service, and search functionality is operating as expected at this time. We will continue to monitor the system closely to ensure ongoing stability. Thank you for your patience.

Read the full incident report →

Started Feb 13, 2026, 06:19 PM UTC· resolved Feb 18, 2026, 12:43 PM UTC

1. investigating Feb 13, 2026, 06:19 PM UTC

   Our operations team has identified an increase in Cloudflare blockages to partners when accessing Koha and Aspen. They are currently investigating and will have an update in one hour.

2. identified Feb 13, 2026, 07:26 PM UTC

   Update: investigation so far has shown that Cloudflare appears to be blocking traffic more aggressively today than in the past, and we are working with Cloudflare on this sudden change in behaviour. If you experience a block, please open a ticket including the following: the public IP of the machine, the RayID provided by Cloudflare, and the query/URL that was trying to be accessed at the time of the block.

3. identified Feb 17, 2026, 02:24 PM UTC

   We are continuing to work with Cloudflare regarding the recent increase in traffic blocks affecting partner access to Koha and Aspen. Cloudflare has confirmed that they are actively investigating the behavior on their side. An internal escalation has been opened with their engineering team to determine what is causing the unexpected increase in blocks. In parallel, they are reviewing our configuration to identify any potential adjustments or workarounds that may help reduce impact while the deeper investigation continues. At this time: - The issue remains under active review by Cloudflare engineering. - We are in direct contact with their Network Security team. - No final root cause has yet been identified. If you experience a block, please continue to open a support ticket and include: - The public IP address of the affected machine - The Cloudflare Ray ID shown on the block page - The full URL/query being accessed at the time This information is critical for escalation and log correlation with Cloudflare. We will provide another update as soon as we receive additional information from Cloudflare engineering.

4. identified Feb 17, 2026, 03:59 PM UTC

   Update on Cloudflare Blocks We are actively working with Cloudflare regarding the recent increase in access blocks affecting Koha and Aspen. Cloudflare has identified that many of the recent blocks are being triggered by their JavaScript Detection system, which is assigning false-positive bot scores. Their engineering team is investigating this behavior on their side. In the meantime, we are seeing indications that some of these challenges may be failing due to browser-side issues (such as cached data or stored cookies interfering with Cloudflare’s JavaScript checks). If you are currently experiencing a block, please try the following: 1. Fully close your browser. 2. Clear cache and cookies for ByWater-hosted services (or clear all browsing data). 3. Reopen the browser and try again. 4. If the issue persists, try an incognito/private window as a quick test. If you are still blocked after completing these steps, please reply with: - The public IP address of the affected workstation - The Cloudflare Ray ID shown on the block page - The full URL being accessed at the time We appreciate your patience while Cloudflare continues their investigation. We will provide additional updates as soon as more information becomes available.

5. resolved Feb 18, 2026, 12:43 PM UTC

   Cloudflare shared the following incident with us, which was resolved at approximately 4:00 pm Eastern on 2026-02-17: https://www.cloudflarestatus.com/incidents/fdvwdt9gy6yc If you see any Cloudflare blocks after 4:00 pm Eastern on 2026-02-17, please try the troubleshooting steps as outlined earlier. If clearing your browser's cache doesn't solve the problem, please send us a ticket.

Read the full incident report →

Started Dec 19, 2025, 09:00 PM UTC· resolved Dec 20, 2025, 12:20 AM UTC

1. investigating Dec 19, 2025, 09:00 PM UTC

   Our operations team has identified an issue with an upgrade process within Aspen. The team has begun implementing a fix, and some impacted partners will already see recovery.

2. resolved Dec 20, 2025, 12:20 AM UTC

   Our team has implemented a fix, and Aspen services should be restored.

Read the full incident report →

Started Nov 18, 2025, 11:24 AM UTC· resolved Nov 18, 2025, 01:46 PM UTC

1. investigating Nov 18, 2025, 11:24 AM UTC

   It is Cloudflare’s turn for a world wide outage (following the footsteps of Amazon and Azure of a few weeks ago). Good news is it doesn’t appear it’s impacting us - if it is please let us know. The CF folks are working on resolving it quickly https://www.cloudflarestatus.com

2. monitoring Nov 18, 2025, 12:16 PM UTC

   We are seeing reports of our partners being impacted by CF's outage. CF let us know they are very close to a fix and we are waiting for them to restore services. They are posting updates here: https://www.cloudflarestatus.com/

3. monitoring Nov 18, 2025, 12:36 PM UTC

   We were able to pause Cloudflare, so your traffic will bypass the CF network and hopefully avoid any outages that may be associated with it.

4. monitoring Nov 18, 2025, 01:02 PM UTC

   After pausing CF, you may be getting an SSL error on your site. This is fine and we can manually renew the cert if you let us know. This occurs because CF stores your cert and by us bypassing CF, CF cannot provide the cert. CF is working on fixing the issue and once its resolved, we will flip back to our protected network and the SSL errors will go away.

5. resolved Nov 18, 2025, 01:46 PM UTC

   Update from CF: Monitoring - A fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal.

Read the full incident report →

Started Oct 29, 2025, 05:49 PM UTC· resolved Oct 29, 2025, 09:18 PM UTC

1. investigating Oct 29, 2025, 05:49 PM UTC

   Our operations team is investigating an issue affecting a subset of our Aspen partners, who are seeing "Could not connect to database" errors. We will have an update within one hour.

2. identified Oct 29, 2025, 06:04 PM UTC

   We believe we identified the core issue affecting partners getting the database issue, which appears to be related to the use of Content Cafe.

3. resolved Oct 29, 2025, 09:18 PM UTC

   For Aspen Partners using Content Cafe, the errors it was receiving were saturating the Aspen Database, using 75% of the database capacity for connections. When we turned off Content Cafe, the issue went away. Shortly after we turned off Content Cafe for all partners, one of our partners shared a message from Baker and Taylor explaining that Content Cafe was having issues as a result of the Microsoft outage today. We recommend keeping Content Cafe off for our partners, but we understand you may want to turn it back on. However, understand that if Content Cafe has future issues or once it is entirely sunset, Aspen services will be interrupted until the feature is disabled again. We understand that the 3rd-party integration outage caused you some inconvenience. Our systems and development teams wanted to ensure we not only identified the root cause but also took proactive steps to prevent similar issues in the future, rather than simply attributing the problem elsewhere. Thank you for alerting us to the issue and sharing the message from your vendors. This is an excellent example of how a large community works together to communicate and resolve a problem.

Read the full incident report →

Started Oct 22, 2025, 12:19 PM UTC· resolved Oct 22, 2025, 08:56 PM UTC

1. investigating Oct 22, 2025, 12:19 PM UTC

   Our operations team has identified an issue affecting a subset of our partners. They are currently investigating and will have an update in one hour.

2. monitoring Oct 22, 2025, 01:42 PM UTC

   The issue has been resolved and we are continuing to monitor our systems.

3. resolved Oct 22, 2025, 08:56 PM UTC

   The issue has been resolved.

Read the full incident report →

Started Sep 24, 2025, 03:14 PM UTC· resolved Sep 24, 2025, 04:52 PM UTC

1. investigating Sep 24, 2025, 03:14 PM UTC

   Our operations team has identified an issue with a subset of hosted Koha. We are working with our upstream provider to remediate, and will have an update within one hour.

2. identified Sep 24, 2025, 03:54 PM UTC

   Our operations team has identified the root issue, we are working with our upstream provider to resolve this issue. We will have another update in one hour.

3. resolved Sep 24, 2025, 04:52 PM UTC

   Partners should now see recovery across the board, we are monitoring closely.

Read the full incident report →

Started Sep 02, 2025, 04:45 PM UTC· resolved Sep 09, 2025, 07:45 PM UTC

1. investigating Sep 02, 2025, 04:45 PM UTC

   Our operations team has seen interruptions to a subset of our partners. We are currently investigating.

2. identified Sep 03, 2025, 04:06 PM UTC

   Update: As a brief explanation: when internet traffic reaches your library’s Koha server, it passes through several network hops and data centers operated by external providers. Occasionally, one of these upstream points experiences problems, and any traffic routed through it can be impacted. You can monitor network health using ThousandEyes. It displays disruptions along the East Coast. We expect this to be resolved soon. In the meantime, if your staff are still experiencing issues, we can temporarily bypass Cloudflare for your staff interface until the upstream connection stabilizes. Please submit a ticket with the subject line SSL handshake errors reroute.

3. resolved Sep 09, 2025, 07:45 PM UTC

   We were able to resolve this issue with our upstream partners last week! We apologize for the inconvenience.

Read the full incident report →

Started Aug 27, 2025, 12:18 PM UTC· resolved Aug 27, 2025, 05:06 PM UTC

1. investigating Aug 27, 2025, 12:18 PM UTC

   Our operations team has identified an issue with keyword searching in Aspen. The issue is currently being worked on, and an update will be posted within 1 hour.

2. identified Aug 27, 2025, 01:44 PM UTC

   The root cause of the issue has been identified and the team is working on a fix. We appreciate your reports, and will continue to update here as progress is made.

3. identified Aug 27, 2025, 02:46 PM UTC

   The team is currently deploying the fix to impacted servers. Updates will continue here as we make progress.

4. identified Aug 27, 2025, 03:45 PM UTC

   Work continues as we push the fix to impacted systems, thank you for your continued reports. Updates will continue to come as we make progress.

5. resolved Aug 27, 2025, 05:06 PM UTC

   We identified a bug with Aspen that led to this issue. We fixed this and rolled out the fix to our partners.

Read the full incident report →

Started Aug 18, 2025, 06:11 PM UTC· resolved Aug 18, 2025, 06:55 PM UTC

1. investigating Aug 18, 2025, 06:11 PM UTC

   A subset of hosted Koha instances may be experiencing some interruption in the staff client (Error 503). We believe we have identified the root cause and are working with our upstream provider to remediate the problem.

2. resolved Aug 18, 2025, 06:55 PM UTC

   The root cause has been addressed with our upstream provider and the issue is resolved.

Read the full incident report →

Started Jul 31, 2025, 02:30 PM UTC· resolved Jul 31, 2025, 07:46 PM UTC

1. investigating Jul 31, 2025, 02:30 PM UTC

   Last night we added additional servers to our infrastructure as part of our continual improvement efforts to enhance reliability and speed. These new servers needed some additional IPs to accept traffic correctly within the environment. We’re working on adding these IPs. If you are experiencing repeated logouts in Koha, please set the SessionRestrictionByIP system preference to "No."

2. resolved Jul 31, 2025, 07:46 PM UTC

   This issue has been resolved. Your SessionRestrictionByIP system preference may be set back to Yes, if you or ByWater set it to No. Please send us a ticket if you continue to experience issues after setting the system preference to Yes.

Read the full incident report →

Started Jul 31, 2025, 02:27 PM UTC· resolved Jul 31, 2025, 07:48 PM UTC

1. investigating Jul 31, 2025, 02:27 PM UTC

   Our operations team has identified an issue affecting the ability to renew items in Aspen, and is working to implement a fix.

2. resolved Jul 31, 2025, 07:48 PM UTC

   This issue has been resolved. Please send us a ticket if you continue to receive errors when renewing items in Aspen.

Read the full incident report →