Innovo Outage History
Innovo is up right nowPingoru has recorded 29 incidents for Innovo: 1 in the last 12 months and 0 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| June 2026 | 1 | 4 of 720 (1%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 0 | 0 of 514 (0%) |
Incidents, newest first
Major June 4, 2026 Broken Google Library 3h 57m Route
Started Jun 04, 2026, 06:07 PM UTC· resolved Jun 04, 2026, 10:04 PM UTC
- investigating Jun 04, 2026, 06:07 PM UTC
We're currently investigating an issue with the Google Maps display in the Customer Portal. It appears to be caused by a broken Google library. This is only affecting Route — all other products are working as expected. We'll keep you posted on our progress.
- identified Jun 04, 2026, 06:13 PM UTC
We've identified the issue and have a fix. We'll be pushing out the update shortly.
- identified Jun 04, 2026, 06:13 PM UTC
We've identified the issue and have a fix. We'll be pushing out the update shortly.
- monitoring Jun 04, 2026, 06:49 PM UTC
A fix has been implemented and we're monitoring the results. Route should be fully functional again. Thanks for your patience!
- resolved Jun 04, 2026, 10:04 PM UTC
This incident has been resolved. We really appreciate your patience and ongoing support!
Critical March 28, 2025 Route and Deliver Web Outage 3d 22h Deliver, Route
Started Mar 28, 2025, 02:08 PM UTC· resolved Apr 01, 2025, 12:42 PM UTC
- investigating Mar 28, 2025, 02:08 PM UTC
It appears we are currently experiencing an outage. We will investigate and let you know as soon as we know more.
- investigating Mar 28, 2025, 02:09 PM UTC
We are continuing to investigate this issue.
- investigating Mar 28, 2025, 02:34 PM UTC
We've pointed our DNS to our backup server so Route and Deliver Web should be coming back online. We are still investigating what caused the outage and will keep you posted.
- investigating Mar 28, 2025, 03:56 PM UTC
The DNS change worked for a bit and then we went down again. We have been on the phone with our database provider and AWS all morning to try to get to the bottom of things. We will keep you posted as we learn more.
- monitoring Mar 28, 2025, 04:33 PM UTC
We have identified the issue to be with MongoDB (our database provider) and are working with their engineers on implementing auto scaling of our cluster to mitigate the issue. This seems to be working for the moment. We will continue to monitor throughout the day. Thank you all for your patience. All of your signatures, photos, comments, etc. were being transmitted to Eclipse during the outage just as normal. The outage was only on the portal side.
- monitoring Mar 28, 2025, 05:57 PM UTC
Unfortunately we are still experiencing issues and connections are intermittent. We will keep you updated.
- monitoring Mar 29, 2025, 12:47 AM UTC
We have finally identified root cause of the outage. In January we made a change to turn on Deliver alerts for everyone by default. Because of this change, this is now our third outage so far this year. This is more than we've experienced in past years. The two previous times we put some new indexes in place and thought the issue was resolved. Unfortunately that was not the case. Today the DB server pegged at 100% CPU and when that happens the application server starts dropping connections and then struggles to reconnect to get a connection to the database. Luckily the connections still went through, however slow so theoretically the portal should still show the stops as delivered, skipped, etc. And like I said previously all signatures, photos, and comments were successfully updated to Eclipse. To fix this today, we increased the RAM on our DB server based on the recommendations from Mongo. On Sunday we will also be creating additional indexes to mitigate any further slowness. In addition, we will be working with Mongo next week to implement further performance recommendations. Again, I am really sorry this happened today and we do really appreciate your patience. We will continue to keep you updated as we learn more.
- resolved Apr 01, 2025, 12:42 PM UTC
We had a great day yesterday! During our peak time (10am-2pm EDT), we averaged about 350 transactions a second and 2,800+ reads per second. Average CPU usage remained steady at 20-40%. In addition to the increase of RAM on Friday, we rebuilt indexes over the weekend which has really seemed to help. We will continue to monitor numbers this week and if we find anything else that needs to be changed to keep up with the load, we will do so. Thank you for your continued patience - we really appreciate you!
Critical February 4, 2025 Route & Deliver Web Down 7h 54m Deliver, Route
Started Feb 04, 2025, 02:00 PM UTC· resolved Feb 04, 2025, 09:54 PM UTC
- investigating Feb 04, 2025, 02:00 PM UTC
We are just receiving word that Route and Deliver Web are currently not functioning. We are investigating now and will provide status updates.
- monitoring Feb 04, 2025, 02:39 PM UTC
We have identified the issue and pushed traffic for now to our backup server. This is a DNS change so it might take a bit for everyone to come back online. You may have to close and re-open your browser to see activity again. Just a reminder, the Deliver app was fully functional during this outage. Unfortunately this was the same issue as what happened last time. We will be making additional adjustments to fix. We will continue to monitor throughout the day and provide additional updates as needed. Thank you so much for your patience. We know how critical these services are and will work to ensure this does not happen again.
- monitoring Feb 04, 2025, 03:28 PM UTC
The system is down again. We are investigating.
- identified Feb 04, 2025, 04:03 PM UTC
We have escalated the issue with our provider and they have responded with some recommendations we can put in place immediately. I will keep updating the status as we know more.
- monitoring Feb 04, 2025, 04:30 PM UTC
We will continue to monitor traffic today and make sure most systems are up and running. Currently the system is back up. We will be working off-hours tonight with our provider to make more permanent changes.
- resolved Feb 04, 2025, 09:54 PM UTC
Closing
Critical January 23, 2025 Deliver Web and Route Outage 3d 22h Deliver, Route
Started Jan 23, 2025, 03:31 PM UTC· resolved Jan 27, 2025, 02:17 PM UTC
- investigating Jan 23, 2025, 03:32 PM UTC
We are currently investigating the issue and will provide status updates.
- monitoring Jan 23, 2025, 04:50 PM UTC
We have identified and fixed the issue and all services should be coming back online. This will be a rolling process as we had to make a DNS change. We have not yet found the root cause so we will continue to monitor the systems throughout the day and provide additional updates as needed. Unfortunately any breadcrumb data that was accumulated in the Deliver app during the time of the outage will not show in Deliver Web. And you may see manifests that were completed during the outage not reflect a completed status in Deliver Web. During the time of the outage, all communication with Eclipse was intact so signatures, photos, comments, etc. were updated as normal. We really appreciate your patience and will let you know as soon as we know more.
- monitoring Jan 24, 2025, 02:29 PM UTC
We still haven't found the actual root cause for the outage yesterday but we did find there was a spike in IOPS which is the standard unit of measurement for the maximum number of read/writes per second. This could have been caused by the portal change implemented this past weekend to turn on Deliver alerts by default for everyone or it could have been caused by a bad query. We are just not sure yet. There were also out of memory issues so we increased the heap size to account for that. We will monitor the IOPS today to see if that needs to be increased as well. Again, we really appreciate everyone's patience yesterday and apologize for the inconvenience. Fortunately operations were only affected for a short time.
- resolved Jan 27, 2025, 02:17 PM UTC
This incident has been resolved. We created a couple indexes to address some slower queries over the weekend and believe that should address this issue moving forward. We appreciate everyone's patience!!