365 Retail Markets incident
GMA - Delays in updating funding transactions
365 Retail Markets experienced a minor incident on August 11, 2022 affecting Global Market Accounts, lasting 4d 10h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 11, 2022, 12:00 PM UTC
Dear Valued Customers, Global Market Accounts (GMA) has been experiencing intermittent delays in updating funding transactions, averaging anywhere between 30 and 60 minutes. This slowness is unexpectedly being caused from network throttling by Amazon Web Services (AWS). Our leadership, development and operations teams are actively working with AWS to determine the root cause to eliminate these intermittent delays. Important Note: No funding data has been lost during the updating delay. We will provide further updates as we progress. Thank you, 365 Retail Markets
- monitoring Aug 11, 2022, 08:31 PM UTC
Dear Valued Customer, Thank you for your patience and understanding while we have been working to resolve the intermittent delayed funding transactions issue. We have been all-hands-on-deck to not only continue operation in your markets, but also find and fix the root cause of the problem. Wednesday night we escalated the issue to Amazon Web Services (AWS) and we implemented the changes they suggested to improve fluctuating load management. We have closely monitored our environments, today, and no new issues have occurred. Additionally, tonight we are making changes to our batch processing size to handle peak periods more efficiently. We will continue to closely monitor the situation and make additional changes to improve our environment's resiliency as required. Thank you, 365 Retail Markets
- resolved Aug 15, 2022, 10:19 PM UTC
Dear Valued Customer, We have been closely monitoring our environments since last week and have seen no new occurrences of delayed funding transactions. All systems are functioning as expected. We apologize for this inconvenience and truly appreciate your partnership and patience. Thank you, 365 Retail Markets