GoTab experienced a notice incident on April 12, 2025, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Apr 12, 2025, 11:00 AM UTC
On 4/5/25, between 2:30 am EDT and 3:10 am EDT, GoTab experienced an outage similar to the prior day's incident. The root cause was the same. The investigation from the prior incident had inaccurately identified a failed database backup process on Amazon Web Services and a high priority ticket had been opened with AWS support. The investigation had been ongoing at the time this second incident occurred. As a result of additional data gathered in this secondary incident from 4/5/25, we were able to accurately diagnose and identify the true root cause of the issue. A newly-introduced bulk payment data processing job had inadvertently caused a spike in the number of real-time notifications that were being initiated from the database to Redis listeners to propagate outward to other consuming services. With this new understanding of the nature of the underlying issue, our engineers immediately deactivated the scheduled task and revised the processing strategy to avoid future similar incidents.