Webshipper experienced a major incident on May 4, 2023 affecting Frontend, lasting 4h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 04, 2023, 07:18 AM UTC
We are currently experiencing internal issues in the system with some shipments. We are working intently on fixing all issues and apologize for the issues at hand.
- monitoring May 04, 2023, 08:11 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 04, 2023, 11:25 AM UTC
All issues related to internal server error and barcodes have been resolved since this morning.
- postmortem May 08, 2023, 06:12 AM UTC
Time stamp: 4th of May 14:00 to 5th of May 09:20 Issue: internal Errors on shipments Description: In an attempt to expand and upgrade the foundations for barcode formatting for all EDI based carrier integrations, we encountered some unexpected issues when changes were made to the barcode creation part of Webshippers API and carriers pool, which had unintended consequences. Due to these changes, duplicated barcodes were sometimes inadvertently generated, leading to Internal server errors on shipments and other errors with shipments such as track & trace issues. Although the faulty code was quickly reverted in the afternoon and successfully resolving the problem for the carrier pool service, a part of the code difference on the API was not reverted correctly on the new build due to an internal error from Bitbucket, behind the build process, creating an out of sync main branch, which caused the incident to persist for longer than intended. Due to the revert on the carrier pool being successful, some of the errors shown through the API were removed leading to false positives late in the afternoon. When discovered next morning, our team worked to address the issue as fast as we could, and the API was fully rolled back to the previous version on the morning of 5th of May, effectively closing the incident. Going forward, we will look into and implement better and stricter testing protocols for barcodes generation to prevent similar issues from arising in the future.