PDF-Generator API experienced a major incident on June 18, 2025 affecting Document Generation Service, lasting 25m. The incident has been resolved; the full update timeline is below.
Affected components
Document Generation Service
Update timeline
- identified Jun 18, 2025, 10:30 AM UTC
We're experiencing an elevated level of API errors due to cache issues. The team is working on solving the issue.
- identified Jun 18, 2025, 10:42 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Jun 18, 2025, 10:51 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 18, 2025, 10:56 AM UTC
Root cause: Redis ran out of memory when a huge base64-encoded PDF got stuck in the document-delivery queue, pushing usage over 90%. The team temporarily upgraded the Redis node to a bigger instance, which cleared the alert, and then deleted the oversized queue key to free up about 1.2 GB.