PNGs and PDFs downloading as blank

Update timeline

1. monitoring Oct 08, 2025, 07:08 PM UTC

   A fix has been implemented and we are monitoring the results.

2. identified Oct 08, 2025, 07:12 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Oct 08, 2025, 07:12 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Oct 08, 2025, 07:18 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Oct 09, 2025, 12:11 AM UTC

   This incident has been resolved.

6. postmortem Oct 09, 2025, 03:16 PM UTC

   ### Timeline On October 8th, at 1:27PM EST we deployed a package upgrade to our backup rendering service which is responsible for generating previews of pdfs and pngs. At 2:06PM EST we routed our production traffic to our backup rendering service. During which time it was monitored noting that the service was succeeding users were able to download pdfs and pngs At 3:01PM EST Knak Engineers were informed that the pdfs and pngs being download were blank pdfs and pngs. At 3:18PM EST Knak Engineers reverted the traffic back to the primary rendering service, which they then confirmed that pdfs and pngs were correctly downloading ### Root Cause An underlying package that our rendering service relies on has implemented a breaking change. It was believed that the package upgrade followed standard semantic versioning which would mean that this was minor change which would not impact production, it was later discovered it does not follow semantic versioning. This was not noticed in our non production environments as functionally from an integration standpoint our rendering service was taking in code and rendering a pdf/png. ### Action Taken * We have rolled back the change and went back to testing the package upgrade * We have documented how to manually end to end test our rendering service * We are working on implementing a fully automated end to end test for our application/rendering service