ProPay experienced a major incident on October 14, 2024 affecting Direct Credit Card and ACH Processing, lasting 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 14, 2024, 06:59 PM UTC
We are aware of an issue that is preventing some transactions from being processed. Our teams are working to resolve the issue as quickly as possible.
- monitoring Oct 14, 2024, 07:07 PM UTC
A fix has been implemented and we are monitoring the results.
- investigating Oct 14, 2024, 07:18 PM UTC
We are continuing to investigate this issue.
- investigating Oct 14, 2024, 07:19 PM UTC
We're still observing some errors and are working on some additional mitigation.
- investigating Oct 14, 2024, 07:35 PM UTC
We are continuing to investigate this issue.
- investigating Oct 14, 2024, 07:36 PM UTC
Processing was restored at 13:23MT and we're still monitoring.
- resolved Oct 14, 2024, 07:36 PM UTC
This incident has been resolved.
- postmortem Oct 28, 2024, 07:15 PM UTC
Regarding this incident from Oct 14 between 12:25 MT - 13:25 MT which caused transaction timeouts: Global Payments support teams identified that several of the servers were in an unhealthy status after maintenance was performed to relieve an increased size of the replication distribution database. This maintenance caused a disconnect between the application layer and the database layer, resulting in the transactions receiving a time out. Resolution: Support teams performed a failover to the secondary server, which restored service. Additional review is being performed to improve monitoring, alerting and response plans for any similar situations in the future.