Innovo Outage History

We have identified the issue and pushed traffic for now to our backup server. This is a DNS change so it might take a bit for everyone to come back online. You may have to close and re-open your browser to see activity again. Just a reminder, the Deliver app was fully functional during this outage. Unfortunately this was the same issue as what happened last time. We will be making additional adjustments to fix. We will continue to monitor throughout the day and provide additional updates as needed. Thank you so much for your patience. We know how critical these services are and will work to ensure this does not happen again.

The system is down again. We are investigating.

We will continue to monitor traffic today and make sure most systems are up and running. Currently the system is back up. We will be working off-hours tonight with our provider to make more permanent changes.

We have identified and fixed the issue and all services should be coming back online. This will be a rolling process as we had to make a DNS change. We have not yet found the root cause so we will continue to monitor the systems throughout the day and provide additional updates as needed. Unfortunately any breadcrumb data that was accumulated in the Deliver app during the time of the outage will not show in Deliver Web. And you may see manifests that were completed during the outage not reflect a completed status in Deliver Web. During the time of the outage, all communication with Eclipse was intact so signatures, photos, comments, etc. were updated as normal. We really appreciate your patience and will let you know as soon as we know more.

We still haven't found the actual root cause for the outage yesterday but we did find there was a spike in IOPS which is the standard unit of measurement for the maximum number of read/writes per second. This could have been caused by the portal change implemented this past weekend to turn on Deliver alerts by default for everyone or it could have been caused by a bad query. We are just not sure yet. There were also out of memory issues so we increased the heap size to account for that. We will monitor the IOPS today to see if that needs to be increased as well. Again, we really appreciate everyone's patience yesterday and apologize for the inconvenience. Fortunately operations were only affected for a short time.