ProPay incident

ProPay System Processing Issue

Notice Resolved View vendor source →

ProPay experienced a notice incident on September 26, 2024 affecting Direct Credit Card and ACH Processing, lasting 2h 1m. The incident has been resolved; the full update timeline is below.

Started
Sep 26, 2024, 03:58 PM UTC
Resolved
Sep 26, 2024, 06:00 PM UTC
Duration
2h 1m
Detected by Pingoru
Sep 26, 2024, 03:58 PM UTC

Affected components

Direct Credit Card and ACH Processing

Update timeline

  1. investigating Sep 26, 2024, 03:58 PM UTC

    We are aware of an issue that is preventing some transactions from being processed. Our teams are working to resolve the issue as quickly as possible.

  2. identified Sep 26, 2024, 04:13 PM UTC

    This issue is related to our DNS provider. We are working with them to address the issue as soon as possible.

  3. identified Sep 26, 2024, 04:46 PM UTC

    Our teams are implementing mitigation efforts to reduce impact while the provider works to address the underlying issue.

  4. identified Sep 26, 2024, 05:30 PM UTC

    We are continuing to work with our DNS provider. They have indicated they are working on implementing fixes and have seen some improvement, but have additional work to carry out to address the underlying issue.

  5. resolved Sep 26, 2024, 06:00 PM UTC

    This incident is considered resolved as of 11:25am Mountain Time. We will await root cause analysis information from our DNS vendor.

  6. postmortem Oct 01, 2024, 03:30 PM UTC

    The issue from 9/26/24 resulting in processing failures was caused by an issue with our DNS provider, who reported resource consumption issues on their system between 09:00MT-11:25MT. They isolated the offending areas of their network and solution and were able to restore service. They are continuing to review and implement updates to their processes to prevent a recurrence of this issue.