Violet experienced a major incident on August 29, 2024 affecting Production API and Sandbox API and 1 more component, lasting 2h 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Aug 29, 2024, 04:15 PM UTC
We are currently monitoring degraded performance when interacting with the Shopify API. There has been a significant increase in 500 Internal Server Errors returned by the Shopify API when performing product syncs and when performing checkout.
- monitoring Aug 29, 2024, 06:26 PM UTC
We've implemented a solution for the failed Shopify Draft Order submission. It appears that Shopify introduced a bug into this endpoint, https://shopify.dev/docs/api/admin-rest/2024-07/resources/draftorder#put-draft-orders-draft-order-id-complete, at around 9am PT where submission requests would fail because Shopify could not read the length of the request body. As there is no request body with this endpoint this is likely not intended behavior. The Violet system will now detect this specific error and switch over to using the Write Order endpoint anytime it occurs. Any Draft Order resource will be deleted to ensure that there is no orphaned/abandoned entities in the merchants system. Full retro to follow.
- monitoring Aug 29, 2024, 06:31 PM UTC
The 500 Internal Server errors from product related endpoints appear to be subsiding.
- resolved Aug 29, 2024, 06:31 PM UTC
This incident has been resolved.