Innovo experienced a notice incident on March 20, 2024 affecting Deliver and Route, lasting 12d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 20, 2024, 02:49 PM UTC
We are currently investigating reports of Route and Deliver being down. We'll keep you updated as we learn more.
- identified Mar 20, 2024, 03:04 PM UTC
Just a quick update - This seems like the same issue we experienced yesterday and we are currently working with AWS to address the issue. Drivers should still be able to download manifests, capture signatures, and upload photos however it may be a little slower than normal. We will update you as soon as we know more!
- monitoring Mar 20, 2024, 03:57 PM UTC
We are still working with AWS on the root cause however our fail over server is now online and seems to be holding strong so far. Deliver Web and Route are back online. We'll send out another update when we have completely resolved the issue. Thank you for your patience!!
- monitoring Mar 20, 2024, 03:59 PM UTC
We are continuing to monitor for any further issues.
- monitoring Mar 20, 2024, 08:40 PM UTC
Our servers experienced a bottleneck today at peak run time (9am-2pm EDT), similar to what happened yesterday. As soon as we noticed the bottleneck, we pushed all traffic to our failover server and recovered within five minutes. Unfortunately we are still not sure what is causing these bottlenecks so for now we will be increasing the compute power on the production server to handle the network spikes and adjusting the load balancing configuration to fine tune the instance. We will keep you updated with our progress and findings. Thank you for your continued patience!
- resolved Apr 02, 2024, 12:50 PM UTC
The increase of compute power along with adjustments to our load balancer configuration has resolved the issue. Everything has been running smoothly for the past week.