BluePay Transaction Data Partially Unavailable

Update timeline

1. identified Jun 19, 2024, 03:52 PM UTC

   A connectivity issue between BluePay database servers caused transaction data from transactions processed 6/18/2024 4:16pm-4:26pm EDT to not be merged to all database servers. Affected transactions are currently not available in the BluePay website and not available from reporting APIs. We are currently working to correct the issue. Once connectivity is restored between the database servers, the transaction data will be merged making it available in the BluePay website and reporting APIs. The transactions will then settle in the next regularly scheduled settlement.

2. monitoring Jun 19, 2024, 04:07 PM UTC

   Connectivity issue has been corrected and data merged. Data from all transactions is now available.

3. investigating Jun 20, 2024, 05:22 AM UTC

   Additional data has been found to be missing. We will be working to correct the database synchronization issue and restore all the missing data.

4. identified Jun 20, 2024, 01:36 PM UTC

   A database server added to the cluster is failing to synchronize with the other database servers. This is causing some transaction and settlement data near the time of the addition to not be available for reporting and for use as tokens. We are working on correcting the synchronization issue and will provide an update when there is an ETA for correction of the issue. Transactions using a token affected by this issue will receive the message "COULD NOT LOCATE ORIGINAL TRANSACTION". They will need to be reattempted after the issue is resolved.

5. identified Jun 20, 2024, 06:01 PM UTC

   An application to synchronize the data across the database servers has been created. We are starting the process to restore the missing data. A completion time is not known yet.

6. identified Jun 20, 2024, 08:18 PM UTC

   The data synchronization issue has caused some transactions to be double settled. After the missing data is restored we will be identifying the double settled transactions and determining the best way to reverse the duplicates.

7. identified Jun 21, 2024, 12:41 PM UTC

   We are continuing to work on a fix for this issue.

8. identified Jun 21, 2024, 03:17 PM UTC

   During this time please refrain from manually refunding transactions appearing as duplicates. This will be resolved once a fix is in place.

9. identified Jun 21, 2024, 04:32 PM UTC

   We are continuing to work on a fix for this issue.

10. identified Jun 21, 2024, 08:11 PM UTC

    Transactions that previously received a "COULD NOT LOCATE ORIGINAL TRANSACTION" error can now be reprocessed. The original transactions have been restored.

11. identified Jun 23, 2024, 06:30 PM UTC

    We are continuing to work on a fix for this issue.

12. identified Jun 23, 2024, 06:31 PM UTC

    Reversal of double settled credit card transactions over $10,000 on First Data processing platforms has begun. Reversal of smaller First Data credit card transactions, credit card transactions on other platforms and ACH transactions are still being planned.

13. identified Jun 23, 2024, 08:36 PM UTC

    Reversals of double settled credit card transactions over $10,000 on First Data processing platforms are pending settlement. Transaction Type is CREDIT for reversal of a SALE or CAPTURE. Transaction Type is SALE for the reversal of a REFUND. The Comment on the transactions is "Reversal of double settlement.". Reversal of double settled credit card transactions under $10,000 on First Data processing platforms is starting now. It is expected to be completed Monday 6/24/2024.

14. identified Jun 24, 2024, 09:52 PM UTC

    We are continuing to work on a fix for this issue.

15. identified Jun 25, 2024, 12:45 PM UTC

    We are continuing to work on a fix for this issue.

16. identified Jun 25, 2024, 05:51 PM UTC

    Duplicate settlements on the BluePay Gateway are currently in the process of being reversed. This may take up to 3 business days to be completed. Please subscribe to status.bluepay.com for updates to the timeline as we have them.

17. identified Jun 26, 2024, 02:12 PM UTC

    Double settled Fiserv ACH Third Party Sender transactions have been reversed. We continue to work on reversals for Fiserv ACH Direct to Bank and credit card transactions. We will continue to provide updates as they become available.

18. identified Jun 27, 2024, 03:02 PM UTC

    We are continuing to correct the remaining double settled transactions. We will continue to provide updates as they become available.

19. identified Jul 01, 2024, 03:32 PM UTC

    We are continuing to correct the remaining double settled transactions. We will continue to provide updates as they become available.

20. identified Jul 01, 2024, 05:57 PM UTC

    Duplicated settlement reversals for all ACH transactions have been submitted. North and Omaha duplicated settlement reversals have also been submitted. We are continuing work through recovery efforts including correcting transaction records. A full impact list will be provided to account managers to be provided upon request in the coming days. We will continue to provide updates as they become available.

21. identified Jul 02, 2024, 01:55 PM UTC

    Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process. Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed. Duplicated credit card settlements on the First Data South processing platform are being worked on. Account managers will have full impact lists available by request in the coming days. We will continue to provide updates as they become available.

22. identified Jul 02, 2024, 04:43 PM UTC

    Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process. Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed. Duplicated credit card settlements on the First Data South processing platform are being worked on. Invalid batch sequence errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue. Account managers will have full impact lists available by request in the coming days. We will continue to provide updates as they become available.

23. identified Jul 03, 2024, 05:01 PM UTC

    Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process. Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed. Duplicated credit card settlements on the First Data South processing platform are in the process of being reversed. Device ID errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue. Account managers will have full impact lists available by request in the coming days. We will continue to provide updates as they become available.

24. identified Jul 08, 2024, 02:09 PM UTC

    Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process. Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed. Duplicated credit card settlements on the First Data South processing platform are in the process of being reversed. Device ID errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue. Account managers will have full impact lists available by request in the coming days. We will continue to provide updates as they become available.

25. identified Jul 09, 2024, 04:15 PM UTC

    Duplicate settlements have been reversed/fixed. We are currently preparing a detailed master impact list for all affected items. We have identified a grouping of Omaha Batches which did not settle on 6/19, We are working to review the pending authorizations and will close these batches once validated. Please continue to monitor account activity. We will continue to provide updates as they become available.

26. identified Jul 10, 2024, 04:43 PM UTC

    Duplicate settlements have been reversed/fixed. We are currently preparing a detailed master impact list for all affected items. We have identified a grouping of Omaha Batches which did not settle on 6/19, We are working to review the pending authorizations and will close these batches once validated. Please continue to monitor account activity. We will continue to provide updates as they become available.

27. identified Jul 26, 2024, 04:16 PM UTC

    Duplicate settlements have been reversed/fixed. Omaha Batches which did not settle on 6/19 are in the process of being manually closed once validated. Please continue to monitor account activity. We will continue to provide updates as they become available.

28. identified Aug 07, 2024, 04:15 PM UTC

    Duplicate settlements have been reversed/fixed. Omaha Batches which did not settle on 6/19 have been reviewed and prepared for settlement. Funding should be expected within the next 48 hours. Please continue to monitor account activity. We will continue to provide updates as they become available.

29. identified Aug 07, 2024, 10:18 PM UTC

    Duplicate settlements have been reversed/fixed. Omaha Batches which did not settle on 6/19 were submitted for funding on 8/5, merchants should see funds processing through their accounts. Please continue to monitor account activity. We will continue to provide updates as they become available, if there are questions or concerns please contact your support team.

30. resolved Jan 27, 2025, 04:29 PM UTC

    This incident has been resolved.