Sticky experienced a minor incident on April 4, 2019 affecting Third Party Integrations, lasting 4d 16h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 04, 2019, 02:18 AM UTC
We have seen high latency in processing transactions against both Inovio and Argus Payment gateways. See the following screenshot: https://take.ms/ntgsw Since about 1:00pm ET on Tuesday, April 2, average response times from Inovio and Argus payment gateways has jumped to 10 seconds from the usual 1-2 second response times. We are continuing to monitor and review this issue. All other payment gateway integrations are running normally. Separately, from about 3:45pm ET to 6:00pm ET on Wednesday, April 3, transactions processed against Argus Payment gateway were failing due to gateway integration changes introduced in today's LimeLight release. For full transparency, you can see the effects of the issue in this screenshot: https://take.ms/k2zYE - however, notice that the transaction failure timeframe is also mixed up with the processing latency issue which is not in our control. Support for 3D Verify was added to the Argus Payment gateway, and a profile migration issue caused the "Site ID" authentication parameter to not be sent in the Argus gateway request, thus resulting in transactions not getting processed during this 2 hour window. We are also reviewing a potential issue regarding refunds on the Argus Payment gateway failing due to an upgrade to Argus/Inovio API version 3.6 (from previous API version 3.3). It may have introduced some incompatibilities with orders originally processed against the old version 3.3 API. Our engineering teams are reviewing this.
- identified Apr 04, 2019, 02:48 PM UTC
We have identified the issue with refunds on Argus Payment gateways failing. Orders processed after 6:00pm on 4/3/2019 will refund normally. Orders prior to 6:00pm on 4/3/2019 were failing with an internal message "Order not found" from the gateway. We have determined that we can run a data patch to resolve the refund transaction compatibility issue introduced with our 4/3/2019 release for Argus Payment gateway transactions. This should be resolved by 12:00pm ET today 4/4/2019 and refunds against Argus Payment orders will work properly. As to the latency issues with both Inovio and Argus Payment gateways, although some transactions are processing normally within 1-2 seconds, there are still transactions taking upwards of 20-30 seconds to process. See screenshots https://take.ms/uVgFd and https://take.ms/iX4dj - we have an open ticket with Inovio support to continue to troubleshoot this latency issue.
- monitoring Apr 04, 2019, 06:13 PM UTC
Inovio has confirmed that the latency issue is due to a connection between Inovio and one of their 3rd party vendors. We are attempting to find out whether this is a specific 3rd party processor, which may explain why some transactions are running normally but others are taking 30+ seconds. Unfortunately this situation is not in LimeLight's control. If this is affecting your business, we recommend that you reach out directly to your Inovio contact to push for a resolution.
- monitoring Apr 04, 2019, 08:27 PM UTC
We have identified some additional Argus Payment gateway refund issues where we are receiving "Order not found" responses. We are currently running another data patch to resolve the issue. Refunds against Argus Payment gateway transactions should be working for all customers in the next 1 hour (by 4/4/2019 5:30pm ET).
- monitoring Apr 05, 2019, 12:38 AM UTC
We have resolved the Argus Payment gateway refund "Order not found" issue and have confirmed with several clients that it is working correctly now. We apologize for any inconvenience this has caused. We have also received confirmation from Inovio support that they have identified the problematic 3rd party that is causing the sporadic latency issues in transaction processing. They are working to resolve the situation soon. We will update this case with any news that we hear from them. If the Argus/Inovio gateway issues are affecting you (seeing auth/sale transactions taking longer than 10 seconds on a regular basis), please contact Inovio support directly so that they are aware this is affecting more of their merchants than they might think.
- monitoring Apr 05, 2019, 02:17 PM UTC
As of this morning, 10:15 AM ET on 4/5/2019, we are still seeing slow response times from Inovio https://take.ms/PFrqd and Argus https://take.ms/Qqg0K payment gateways. We will continue to update you as we get more information on their resolution efforts.
- monitoring Apr 05, 2019, 09:23 PM UTC
As of 3:20pm ET on 4/5/2019, the Inovio https://take.ms/XWPbd and Argus Payment https://take.ms/rx7kP response times are back to normal. We have not received positive confirmation from Inovio that the issue is permanently fixed but our data indicates response times back to normal 1-2 second latency.
- resolved Apr 08, 2019, 07:00 PM UTC
This incident has been resolved.