2024-10-12 - Permission Issue with Shopify Landed Cost API

Update timeline

1. investigating Oct 14, 2024, 09:21 PM UTC

   We are currently investigating this issue.

2. identified Oct 14, 2024, 09:22 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 14, 2024, 09:23 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 14, 2024, 09:23 PM UTC

   This incident has been resolved.

5. postmortem Oct 14, 2024, 09:24 PM UTC

   **What products were affected and what was the impact?** On **Saturday, October 12, 2024, at 22:30 UTC**, we experienced an outage that impacted the ability of some Shopify merchants to retrieve landed cost quotes for international orders. This outage lasted until **Sunday, October 13, 2024, at 21:33 UTC**, during which time affected merchants may have been unable to display accurate landed cost calculations to their customers, potentially affecting international checkouts and purchase completions. The root cause of the outage was an API permission issue that was introduced during a routine deployment. Unfortunately, the synthetic tests for the Landed Cost API flow didn’t immediately flag the issue, which caused an extremely uncharacteristic delay to the resolution. It's important to note that this issue was caused by a very unique set of circumstances. We have just finished an extremely complex and expansive migration project to move all of our merchants to a robust and secure new system. This issue was directly caused by processes related to this now complete migration. Impact: critical ‌ **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | October 12, 2024 | 22:30 UTC | | October 13, 2024 | 21:33 UTC | ‌ **How was the issue detected?** The engineering team noticed a warning of failed quotes in the log files. ‌ **What functionality was affected?** All landed cost requests during the outage failed for affected merchants. ‌ **What problems did this cause?** Shoppers of affected merchants were unable to get landed cost quotes, and in many cases complete their checkout. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** Once the issue was identified, our team acted swiftly to permission issue. After thorough testing, we confirmed the API was fully restored at **21:33 UTC** on Sunday. ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Expanded test coverage**: We are enhancing our synthetic tests to include additional token and permission checks, ensuring that potential issues like this are caught earlier. * **Improved log classifications**: We’re updating our logging processes to better distinguish between warnings and errors, which will enable faster detection and resolution of similar problems in the future.