Paazl shipping labels not being generated

Update timeline

1. investigating Jul 09, 2024, 10:25 AM UTC

   Paazl is currently experiencing an issue with generating labels. We're looking into it and will post updates as soon as we have more information.

2. monitoring Jul 09, 2024, 10:40 AM UTC

   We have identified the issue and rolled a fix.

3. identified Jul 09, 2024, 01:04 PM UTC

   The issue has not been completely resolved in every instance. We're currently looking into it again.

4. identified Jul 09, 2024, 01:25 PM UTC

   The problem has been identified and we're currently initiating a rollback.

5. monitoring Jul 09, 2024, 01:53 PM UTC

   We performed a rollback and are currently monitoring the performance.

6. resolved Jul 10, 2024, 09:09 AM UTC

   This incident has been resolved.

7. postmortem Jul 10, 2024, 09:09 AM UTC

   Yesterday we experienced degraded performance due to two different issues. First, A server upgrade ended up in us losing a previously configured time zone, which was essential for some of our platform functionality. This caused issues with printing labels. This was solved within the hour. This fixed the label issues for most customers. Setting our status page to monitoring. During the monitoring phase, a second issue was reported by a few customers around 13:15. The SOAP generateLabel response was changed to have a namespace prefix ns2:. The introduction of this non-blank namespace prefix in our SOAP responses caused issues with some integrations using XML parsers not equipped to deal with namespace prefixes. This was fixed with a rollback around 15:30.‌ Unfortunately, this is an issue undetectable by heavy testing, as this is coming from OMS/WMS specific settings. We are currently working on a fix to suppress the namespace and intent to roll out this release today during the afternoon. However, to avoid similar issues we also heavily advise our customers to update their parsing mechanism for scalability.