Checkout - investigating incident

Update timeline

1. investigating Aug 30, 2026, 05:03 AM UTC

   We are currently experience downtime on our Peach Payments checkout, whereby cardholders are unable to checkout.

2. resolved Aug 30, 2026, 05:26 AM UTC

   This incident has been resolved - an RCA will be shared by Monday 31 August 2026

3. postmortem Sep 01, 2026, 05:51 AM UTC

   **Checkout Outage** **Issued: 31 August 2026** **What was affected?** On Sunday 30 August 2026, checkout requests failed or timed out from 04:03:46 until 07:19:30 SAST \(3 hours 16 minutes\). Shoppers attempting to complete payment via standard Checkout or Checkout v2 experienced failures and hung requests until the request timeout was reached. **What was the root cause?** Checkout maintains session and merchant data in a managed cache. During a scheduled weekly provider maintenance window \(opened 04:00:00 SAST\), the provider replaced the primary node in the cluster and completed a failover to the replica at 04:03:46.312 SAST. Our services were unable to detect the stale connection and remained connected to the failing node. Warm application instances retained hanging connections to the replaced node, causing every new checkout request to reuse the stale connection and wait out the full request timeout. The service interruption persisted until we manually failed over to a healthy node, which restored services at 07:19:30 [SAST.At](http://SAST.At) 08:01:35 SAST we attempted to fail over to an alternative cache service which was assumed to be more reliable. However, this cache service was incompatible with our cache client. Warm application instances were still reusing the original cluster connection so checkout requests succeeded for several minutes after the switch to the alternate cluster. At 08:49:04 SAST we saw intermittent errors as our application instances started using the alternate cluster causing a second outage. At this point we swapped back to the original cache service which began restoration of traffic at 09:12:00 SAST with all traffic succeeding by 09:13:39 SAST.Furthermore, an observability gap existed where initial alerts failed to page the relevant teams, leading to delayed incident response. **Resolution and Remediation** **Incident Timeline & Response:** **04:00:00 SAST**: Provider maintenance window opened. **04:03:46 SAST**: Provider failover completed; first write errors landed on replaced node. **04:03:46 SAST**: First failed request to checkout v1 **04:03:49 SAST**: First failed request to checkout v2 **04:07:01 SAST:** SEV-1 Alarm fired for escalating checkout issues **06:08:00 SAST**: Support reported a spike in Internal server errors **06:19:00 SAST**: Teams started investigating the cause of the outage **07:12:40 SAST**: Temporary manual failback to the remaining healthy node was initiated. **07:19:30 SAST**: Last checkout requests timed out. Service restored **07:20:00 SAST**: Latency returned to normal \(~385 ms\) with zero remaining errors. **08:01:35** **SAST**: Attempt to reconfigure a more reliable cache cluster **08:49:04 SAST**: Observed an increase in checkout errors **09:12:00 SAST**: Revert to original cache service **09:13:39 SAST**: Last checkout requests timed out. Full service restored ‌ **Mitigation Actions** **Stale connection detection**: Configure explicit timeouts on our cache client so dead node connections fail quickly rather than exhausting the full request duration. **Connection Pool Recycling**: Ensure connections are closed or rebuilt on errors/timeouts so subsequent requests establish new connections to the active primary node. **Improved alerting**: We are implementing more comprehensive system monitoring to detect checkout-related errors faster. **If you have any questions concerning the content of this RCA, please open a case via** [**[email protected]**](mailto:[email protected]) **Alternatively, please contact your relevant Account Managers for alternative methods of contacting emergency support.**