Subbly incident
Emergency maintenance affecting checkout and API
Subbly experienced a major incident on November 11, 2024 affecting Checkout and API, lasting 1h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 11, 2024, 06:03 PM UTC
Our team has identified one of our servers which handles a critical job queue is underperforming. This job queue is required for the API which the checkout depends on is underperforming. We're performing a hotfix which will require 15 minutes of downtime affecting the checkout. We're preparing for this downtime just now. We will update here when the checkout is officially offline. We have already planned a permanent fix for this essential componenet to ensure stable performance for BFCM and the holidays, which will be actioned after the hotfix. Sorry for any inconvenience. We'll be right back.
- identified Nov 11, 2024, 06:20 PM UTC
Putting into maintenance mode now. ETA 25 minutes from now.
- identified Nov 11, 2024, 06:49 PM UTC
Unfortunately the hot fix is proving tricky, we're unable to successfully upgrade the server directly, we believe the issue stems from a physical hardware issue. We've started working on the permanent fix immediately and will continue to try finish the hotfix. New ETA is 1 hour. Apologies for any inconvenience.
- identified Nov 11, 2024, 06:56 PM UTC
Good news hotfix is underway, we should have the checkout and API fully back online in 5 minutes
- resolved Nov 11, 2024, 07:04 PM UTC
And we're back to 100% online. API and checkout included. We will continue working on the more robust solution to mitigate a similar situation happening again. The issue was related to either physical hardware failure by our supplier, or an overloaded node on the server. Sorry for any inconvenience caused.