PDF-Generator API incident

Document Generation Delays

Major Resolved View vendor source →

PDF-Generator API experienced a major incident on January 9, 2021 affecting Document Generation Service, lasting 2h 38m. The incident has been resolved; the full update timeline is below.

Started
Jan 09, 2021, 09:41 AM UTC
Resolved
Jan 09, 2021, 12:19 PM UTC
Duration
2h 38m
Detected by Pingoru
Jan 09, 2021, 09:41 AM UTC

Affected components

Document Generation Service

Update timeline

  1. investigating Jan 09, 2021, 09:41 AM UTC

    Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.

  2. investigating Jan 09, 2021, 10:14 AM UTC

    We are continuing to investigate this issue.

  3. identified Jan 09, 2021, 11:06 AM UTC

    We are receiving more document generation request than usually and our auto-scaling processes failed. Due to high CPU load we are currently adding new PODs manually to service all the incoming requests.

  4. monitoring Jan 09, 2021, 11:34 AM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Jan 09, 2021, 12:19 PM UTC

    This incident has been resolved.