Jenji incident

Elevated Vault Errors

Resolved after 13 hours, on December 28, 2021.

postmortem Jun 13, 2022, 03:49 PM UTC

December 28th, 7:00pm - our supplier for legal e-archiving in France deployed a security fix to prevent log4shell exploitation, and the fix wrongly flagged a large number of uploads that resulted in “Connection reset by peer \(Write failed\)“ errors. December 28th, 7:35pm - A ticket was opened by the Jenji Engineering team, with supplier number INC-27857. December 28th, 8:04pm - the errors were acknowledged by our supplier, and work started on their end on reworking the security fix. December 30th, 2:00pm - the updated security fix was successfully deployed in production to their systems. December 30th, 11:23pm - the queued files were successfully all synced and traffic went back to normal.

Started
Dec 28, 2021, 06:00 AM UTC
Resolved
Dec 28, 2021, 07:00 PM UTC
Duration
13h
Detected by Pingoru
Dec 28, 2021, 06:00 AM UTC

Update timeline

  1. resolved Jun 13, 2022, 03:45 PM UTC

    We're experiencing an elevated level of Vault France errors and are currently looking into the issue. No data is at risk.

  2. postmortem Jun 13, 2022, 03:49 PM UTC

    December 28th, 7:00pm - our supplier for legal e-archiving in France deployed a security fix to prevent log4shell exploitation, and the fix wrongly flagged a large number of uploads that resulted in “Connection reset by peer \(Write failed\)“ errors. December 28th, 7:35pm - A ticket was opened by the Jenji Engineering team, with supplier number INC-27857. December 28th, 8:04pm - the errors were acknowledged by our supplier, and work started on their end on reworking the security fix. December 30th, 2:00pm - the updated security fix was successfully deployed in production to their systems. December 30th, 11:23pm - the queued files were successfully all synced and traffic went back to normal.

Your product runs on more than Jenji

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →