Sync experienced a major incident on December 7, 2022 affecting Windows Desktop and Links and 1 more component, lasting 2d 21h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 07, 2022, 07:45 PM UTC
We are currently experiencing a partial outage impacting a subset of users, resulting from a regularly scheduled system update earlier this week. As a precaution, user accounts impacted have been temporarily set to maintenance mode (read-only). Our engineering team is working to restore services as quickly as possible. Impacted users will see a notification in the Sync web panel, and will not be able to add new files until the issue is resolved. Impacted users may also see unexpected errors or duplicate files. It is important to note that your Sync account and files remain safe and secure. This is not a security incident. We will continue to keep you updated as news becomes available, including when we expect service to be fully restored. Thank you for your patience.
- identified Dec 07, 2022, 07:47 PM UTC
We are continuing to work on a fix for this issue.
- identified Dec 08, 2022, 02:01 PM UTC
We understand the urgency of getting service resumed. An integrity check of impacted user accounts has been running throughout the night and into this morning (EST), and is almost complete. This is the final step in getting this issue resolved. We anticipate a gradual restoration of service within the next few hours, and we will be able to provide more frequent updates as things progress. Thanks again for your continued patience on this.
- identified Dec 08, 2022, 03:31 PM UTC
Our engineering team has confirmed that the integrity check of impacted user accounts is complete. We are now running additional checks to test functionality and expect to start restoring service soon.
- monitoring Dec 08, 2022, 05:41 PM UTC
Our engineering team has completed testing and is now restoring services. This will be a gradual process. As your applications connect you may see intermittent connection errors as other impacted users come back online. We are extremely appreciative of your patience as we continue to work towards fully resolving this issue, and we will continue to monitor and provide updates.
- monitoring Dec 08, 2022, 11:27 PM UTC
Our engineering team continues to gradually restore services so all impacted users can resume syncing their files. Resumed service will remain slower than usual while we continue to monitor performance. Thank you for your patience as we work towards fully restoring services
- monitoring Dec 09, 2022, 01:50 PM UTC
We have made significant progress in restoring services, with the majority of impacted user accounts now successfully syncing. Speeds are returning to normal for most users as services come back online. We have also identified a sub-set of user accounts that continue to experience syncing issues, and we are actively working to restore service to these accounts. We will provide additional updates on the progress of this as updates are relayed to us from our engineering team.
- identified Dec 09, 2022, 05:35 PM UTC
Our engineers continue to work towards fully restoring services for you. A required step to improve your connection speeds and Syncing capabilities requires that impacted users go offline for a short window. During this time you will see error messages and be unable to share, upload or download content. We will update the status page when services are back online.
- monitoring Dec 09, 2022, 06:05 PM UTC
Our team of engineers continue to make progress in restoring service to impacted users. A subset of users are continuing to experience syncing issues, and we are actively working to restore service to these accounts. We will provide additional updates on the progress of this as updates are relayed to us from our engineering team.
- resolved Dec 10, 2022, 05:22 PM UTC
Service is now restored, with the majority of impacted user accounts successfully syncing. Next steps: Our Customer Service team is currently working through a backlog of inquiries related to the outage. It may take a few more days for our team to reply to everyone. We are working on it and you will hear from us. If you're still experiencing syncing issues with your account please open a new ticket here: https://www.sync.com/support/ For the past several days our team has been focused on restoring service. With service restored we are now actively working on server side improvements to mitigate further service disruptions, including a top-down review of all actions leading up to this event. We understand the impact this has caused, and we will provide further updates in the coming days. Thanks again for your patience.