Togetherwork incident

[SEV-3] Settlement/Comprehensive File Delay

Minor Resolved View vendor source →

Togetherwork experienced a minor incident on April 28, 2024 affecting File Import Process - Internal and File Import Process - External, lasting 1d 9h. The incident has been resolved; the full update timeline is below.

Started
Apr 28, 2024, 12:10 PM UTC
Resolved
Apr 29, 2024, 09:39 PM UTC
Duration
1d 9h
Detected by Pingoru
Apr 28, 2024, 12:10 PM UTC

Affected components

File Import Process - InternalFile Import Process - External

Update timeline

  1. investigating Apr 28, 2024, 12:10 PM UTC

    Regular settlement files, comprehensive reports and merchant settlement files will not be available by 9:00 a.m. eastern this morning. We will send an update as soon as we have more information and when the files are available.

  2. identified Apr 28, 2024, 05:37 PM UTC

    We believe we identified the problem and it appears to be on ProPay's end. We engaged ProPay about the issue and they are beginning to investigate. This incident likely affects comprehensive reports from yesterday (4/27) as well as today (4/28). As we learn more, we will provide you with updates.

  3. identified Apr 28, 2024, 10:20 PM UTC

    ProPay identified the cause of the issue. Teams are working to correct it and provide missing/corrected reports. We will continue to keep you informed as new information is known.

  4. identified Apr 29, 2024, 01:08 AM UTC

    ProPay is beginning to deliver missing/corrected reports but it will likely take another few hours to finish generating and delivering all of them. Once all files are delivered, they will need to be processed on the Togetherpay side. We will keep you posted as this incident progresses.

  5. identified Apr 29, 2024, 03:59 AM UTC

    We are still waiting on ProPay to deliver files. This incident will carry over until tomorrow, Monday 4/29. We will post updates and things progress.

  6. identified Apr 29, 2024, 12:46 PM UTC

    This morning's files (regular settlement files, comprehensive reports and merchant settlement files) are accurate and available. Teams are still working to re-export comprehensive reports from yesterday (4/28) and Saturday (4/27). It will likely be later this evening before corrected reports for those two days are available. Regular settlement files and merchant settlement files from 4/27 and 4/28 are and have always been accurate.

  7. resolved Apr 29, 2024, 09:39 PM UTC

    All files from ProPay were received and processed. Any missing transactions that should have appeared on 4/27 and 4/28 comprehensive reports are included in today's (4/29) comprehensive. This incident is resolved.