TrueCommerce experienced a major incident on November 19, 2024 affecting Transaction Manager and Pack & Ship and 1 more component, lasting 7h 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 19, 2024, 02:48 PM UTC
We are currently investigating an issue that's introducing intermittent latency with login and some pages loading with TrueCommerce Foundry.
- identified Nov 19, 2024, 03:00 PM UTC
Our data center provider is experiencing an issue with connectivity. Our team is actively working with them on resolution.
- identified Nov 19, 2024, 04:04 PM UTC
We are continuing to work on the issue and network transaction is also seeing latency.
- identified Nov 19, 2024, 04:45 PM UTC
We are continuing to work on a resolution for the issue.
- identified Nov 19, 2024, 06:33 PM UTC
We are continuing to work on a fix.
- identified Nov 19, 2024, 06:47 PM UTC
We are continuing to investigate. Customer will experience latency navigating the Transaction Manager application and transaction processing with TCNet and ecUtopia. TrueCommerce’s technical team is aware of the issue and working to resolve the issue.
- monitoring Nov 19, 2024, 07:58 PM UTC
The latency issue has been resolved. The team is validating that transaction processing has been fully restored.
- monitoring Nov 19, 2024, 07:59 PM UTC
We are continuing to monitor for any further issues.
- identified Nov 19, 2024, 08:41 PM UTC
We are still experience issues with latency within TrueCommerce Foundry application. The team is still in process of restoring all services.
- identified Nov 19, 2024, 08:44 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Nov 19, 2024, 09:34 PM UTC
Transaction processing has been restored. Inbound transactions will experience some delay as processing catches up. The team is currently monitoring.
- resolved Nov 19, 2024, 10:42 PM UTC
This incident has been resolved.
- postmortem Nov 20, 2024, 09:57 PM UTC
* The TrueCommerce incident began at approximately 9:30 a.m. ET on November 19th resulting in intermittent performance degradation of processing transactions on the [TC.net](http://TC.net) network platform and within the TrueCommerce Foundry platform and Transaction Manager application * The root cause of the issue was related to an unplanned incident within a TrueCommerce technology vendor’s internal networking infrastructure * TrueCommerce’s technology vendor resolved the issue at approximately 1:40 p.m. EST * Following the resolution of technology vendor issue, it was determined that a full recycle of TrueCommerce’s core platform services was necessary to ensure all services were restored to full health * Complications were encountered during the platform recycle and startup process which delayed full restoration of services until approximately 4:30 p.m. ET * All platform services were confirmed to be fully restored at approximately 4:30 p.m. EST * All queued transactions were cleared following full restoration of services