Paazl experienced a minor incident on November 25, 2024 affecting Paazl Live Webservice and Paazl Live Web Application, lasting 19h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 25, 2024, 06:35 PM UTC
We are noticing issues with our Web Application. We have identified the issue and are performing corrections.
- monitoring Nov 25, 2024, 06:53 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Nov 25, 2024, 07:42 PM UTC
We have noticed a potential issue with labels that were requested during the time period of this issue. The barcode may not have been pre-registered at the carrier. Please verify in your carrier portal whether the barcodes are known to them. If they are not, then please consider relabeling the packages.
- resolved Nov 26, 2024, 01:37 PM UTC
The fix of yesterday has resolved the issues. Please see the further timeline and impact below. - Timeline 18:57 Paazl monitoring picks up issue with cache cleaning mechanism resulting in partial outages 19:00 Customers begin reporting issues with labels and webapp login 19:20 IT Teams finds the issue 19:31 IT team implements fix 19:39 First customers notice improvements 19:51 IT team confirms that fix has worked for all customers 19:52 Statuspage updated to reflect fix - Technical impact The cache cleaning mechanism that we rely on throughout our platform is set to evict old cache entries whenever it’s full. In this instance it did not do so. We have identified the issue in this mechanism and fixed it. The load testing that we performed in preparation for Black Friday were volume tests, performed over a specific period of time. This kind of stress test does not replicate the cache filling of a full day of volumes. The most significant impact of this issue, related to the generation of shipping labels, and the logging into our webapp. During the affected period, we noticed a potential issue with labels that were requested. The barcodes created may not have been pre-registered at the carrier. We advise you to verify in your carrier portal whether the created barcodes are known to them. If they are not, then please consider re-labeling those packages. We remain confident in our preparation for this peak season and the steps we have taken to make it a success