Deadline Funnel experienced a major incident on February 24, 2021, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Feb 24, 2021, 01:07 AM UTC
We experienced a service degradation in the Deadline Funnel admin from 1:30pm EST to 7:30pm EST on 2/23/21. We apologize for the inconvenience this caused. The issue was related to an underlying database storage issue, but was limited in scope. The Deadline Funnel admin (app.deadlinefunnel.com) was the main service impacted - all other services continued to work as expected during this time. Services that continued working during this time include webhooks, email links, email timers, floating bars, and Portal. The root cause has been identified and resolved. The issue occurred because of a large temporary file that was created by a secondary process in an application database. This large temporary file did not get released automatically and required manual intervention to remove. We are also actively working on implementing additional safeguards and upgrades to prevent this from happening again.