Orderful incident
Legacy Communication Services Delayed Processing
Orderful experienced a notice incident on January 4, 2023 affecting Trading Partner Delivery Services, lasting 8h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 04, 2023, 02:13 PM UTC
Relationships using legacy AS2 and FTP communication channels are experiencing delivery challenges. We are investigating.
- identified Jan 04, 2023, 02:32 PM UTC
The cause of the issue has been identified. Short term mitigation steps are being attempted
- identified Jan 04, 2023, 03:07 PM UTC
Transaction processing delays appear to have peaked and are returning to standard times. Once they are back any failed data will be retried.
- monitoring Jan 04, 2023, 04:17 PM UTC
Resolution in place. Legacy communication channel throughput is fixed so processing continues to catch up. Native communication channels unaffected.
- monitoring Jan 04, 2023, 05:59 PM UTC
Legacy communication channels continue to progress. As time outs are exceeded it may generate failure e-mail notifications. Those may be false positives as the transaction is successfully delivered with a delay. We recommend waiting 60 minutes to verify the delivery failed. We are currently attempting to give the legacy infrastructure more resources to process the back log quicker.
- monitoring Jan 04, 2023, 07:10 PM UTC
Processing catch up has now been able to catch up from "both sides" meaning the oldest affected transactions on legacy comm channels and the newest are being successfully updated and it's working to the middle. This should greatly decrease/stop the false positive e-mail notifications and processing continues.
- resolved Jan 04, 2023, 10:50 PM UTC
All back logged transactions have caught up.