BCH Digital experienced a minor incident on June 14, 2024 affecting 💳 Payments and 💻 Web Portal, lasting 2d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 14, 2024, 08:13 AM UTC
We are currently investigating errors on our Virtual Terminal and Payment Link services when trying to take payments
- investigating Jun 14, 2024, 08:48 AM UTC
We are continuing to investigate this issue.
- identified Jun 14, 2024, 09:33 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jun 14, 2024, 09:43 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 17, 2024, 07:17 AM UTC
This incident has been resolved.
- postmortem Jun 17, 2024, 08:27 AM UTC
The issues experience on Friday were caused by an AWS problem which happened between Tuesday 11th June 06:28 BST and Thursday 13th June 15:16 BST . By the time any issues were reported by clients on Friday 14th June, AWS had cleared the fault their side and so what was experienced was the aftermath of the problem, which made locating the root cause of it more difficult as there were no active issues in AWS to explain this unexpected behavior. The first report was logged to BCH on Friday 14th June 08:54 at which point both Support and Technical Teams began working on this issue. The fault was intermittent and through investigation teams found the root cause of the issue to be down to stickiness on the AWS load balancer \(which ensures when a particular browser session is in progress the requests get routed to the correct server\). This meant requests were not always being routed correctly. To resolve the issues as quickly as possible, whilst further investigation work continued, a reconfiguration was put in place at Friday 14th June 10:21 BST which restored normal functionality. We will continue to run on the reconfiguration used on Friday to allow for any cached cookies to expire and then return to the BAU configuration later in the week. In light of these issues we have also engaged our AWS account manager to understand how we can get better visibility of any issues as they occur to ensure we can respond appropriately. We thank you for your patience whilst we looked into this issue and any inconvenience it has caused you. For reference the information on the fault from AWS is below: _Between June 11 10:28 AM and June 13 7:16 AM PDT, we experienced an issue in the EU-WEST-2 Region where cookies generated by some Application Load Balancers returned a new attribute. This caused some clients to store two different versions of the same cookie name, affecting stickiness and authentication features. The issue has been resolved and the service is operating normally._ _If your end-user is still impacted it is because some clients will continue to send cached cookies. You can fix this by requesting your users to clear the cookies on their clients \(such as browsers\)._