DocNetwork experienced a critical incident on June 10, 2019 affecting Application, lasting 37d 14h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 10, 2019, 01:02 AM UTC
We are experiencing an increase in notifications being sent from our application, and our notification queue is taking longer than expected to send messages as a result. Our Development Team has identified the issue, and is working on a fix. We understand the importance of quick communication, especially at this time of year, and apologize for the inconvenience this is causing you and your families.
- identified Jun 10, 2019, 05:27 PM UTC
While our standard notification queue continues to send messages in chronological order, our Development Team created a script this morning to manually push through high-priority notifications. The Development Team is also working on a larger fix that we believe will solve this issue moving forward. We will provide additional details on timeline for the larger fix once we have them. Please reach out to your Account Manager if you have any urgent notifications that are delayed, and we will do our best to move them to the front of the queue.
- identified Jun 10, 2019, 07:19 PM UTC
Our Development Team is running the script to manually push through all notifications, and we expect the entire backlog of unsent messages to be cleared within the next few hours. We will continue to run this script manually until the larger fix is released.
- monitoring Jun 11, 2019, 02:24 PM UTC
The backlog of unsent messages has been cleared, and our Development Team will continue to monitor the queue while the larger fix is worked on. We have made significant progress on improving our notifications infrastructure, and expect to QA the updates later today before we release any changes to our production environment.
- monitoring Jun 13, 2019, 01:50 PM UTC
Our Development Team continues to run the script to manually push through notifications, and the backlog continues to remain cleared. You should notice only a minimal delay when messages are sent out, if any delay at all. Our updated notifications infrastructure is undergoing QA and we anticipate a release soon.
- resolved Jul 17, 2019, 03:18 PM UTC
After careful QA and continuous monitoring, our updated notifications infrastructure has been released. This issue is now resolved, and all notifications are being sent out in a timely manner.