Syncrofy experienced a minor incident on September 18, 2019 affecting Document Submission Service, lasting 2d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 18, 2019, 12:09 PM UTC
We are currently experiencing delays in document processing. We are working to remediate the issue and will update as we progress.
- identified Sep 18, 2019, 04:27 PM UTC
We have identified the issue and are currently progressing through the queued documents. We are working on an ETA for when this process will be complete and will communicate as soon as we have one.
- monitoring Sep 18, 2019, 09:46 PM UTC
We have continued to steadily progress through the queued documents and expect to catch up overnight tonight. We anticipate no data loss, but will confirm this once we are caught up. Thank you for your patience. As mentioned in the last update, we have identified the root cause for this issue, and have since identified some improvements we can make to improve efficiency in our document processing. We have begun steps to make these improvements and will communicate updates as we progress.
- monitoring Sep 19, 2019, 01:53 PM UTC
Unfortunately we did not make the progress we expected processing the queued documents last night but we are still steadily working through the remaining queue. We are investigating why the system did not finish processing as quickly as we expected and will update with an ETA shortly. Thank you for your patience.
- monitoring Sep 19, 2019, 11:10 PM UTC
Over the past several hours the team has identified and resolved an issue that was impacting the speed in which Syncrofy was indexing documents and files. This meant that even through documents were in the system and persisted, they could not be found via search or filtering because there was a delay in indexing. We can confirm that Syncrofy has been processing and indexing data at normal speeds for several hours now. We are confident no data was lost, however the team has been focusing our efforts on ensuring all data potentially affected during this disruption is properly indexed. To be safe we are performing reindexing on the full week’s worth of data. We expect that this process will be completed by midday tomorrow. We will continue to update the progress on this effort until it is finished. We apologize for the inconvenience and thank you for your patience during this disruption.
- monitoring Sep 20, 2019, 01:59 PM UTC
We are still on track to complete the reindexing by midday. We expect it to be complete around 1PM. The system continues to operate normally (all new data continues to be processed and indexed as expected). We will provide a final update in a few hours when the data integrity efforts are complete.
- resolved Sep 20, 2019, 06:23 PM UTC
All reindexing has completed and Syncrofy is fully operational. We have ensured that any data potentially affected by this disruption is now available via search and filtering. We will be conducting a postmortem on this incident including a root cause analysis and steps we will be taking to prevent something like this in the future. We will publish the postmortem here. Thank you again for your patience.
- postmortem Sep 25, 2019, 05:46 PM UTC
Thank you again for your patience during last week’s disruption. We apologize for any inconvenience it may have caused. ### Here’s what happened: As detailed on [status.syncrofy.com](http://status.syncrofy.com), our team discovered performance issues in document processing on the morning of Wednesday, September 18th. During this time, we took immediate steps to troubleshoot the system and ensure that no data loss occurred. Our team identified an inefficiency in our code that was introduced on Monday, September 16th as the root cause. This bug only manifested after a rare scenario was encountered in production which resulted in an exponential increase in indexing operations, resulting in degraded performance in Syncrofy. ### Here’s what we did: The code for Syncrofy was rolled back to the previous version and systems returned to normal. We took measures to ensure complete data integrity throughout Syncrofy, which took longer than expected, but confirmed there was no data missing from the system. We have since patched the issue and have added additional measures to prevent future occurrences, including additional: • Logging to aid in troubleshooting • Fail safes to prevent similar issues • Monitoring capabilities to ensure data integrity ### Here are extra steps our team is taking: • While our existing monitoring notified our team of these issues quickly, we are expanding this to account for more hypothetical situations. • Perform an analysis of our code and introduce further improvements to processing, indexing, and data resilience throughout Syncrofy. • Update our reprocessing and reindexing mechanisms to ensure system integrity more quickly.