Total Synergy incident
Synergy Cloud - Platform performance issues.
Total Synergy experienced a minor incident on November 7, 2022 affecting Synergy App, lasting 3h 24m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 07, 2022, 12:12 AM UTC
We're investigating multiple reports of platform slowness this morning, 07/11/2022 (11:10 AM AEST) Our engineering team are investigating this with high prioirty and we will provide updates as they become available. We're incredibly sorry for any disruption this is causing you today and endeavour to have this resolved as quickly as possible.
- identified Nov 07, 2022, 12:17 AM UTC
Our engineering team have idetified the problem causing the current platform slowness in Synergy and will be applying a fix within the next 10 minutes. There will be downtime of 5-10 minutes from 11:25 AM AEST, we will continue to provide updates as they become available.
- identified Nov 07, 2022, 01:36 AM UTC
Our Engineering team have made some changes to resolve the platform slowness, however while we have seen some improvement on speeds, we're still getting some reports of slowness, and Global Search is still not loading anything at all. They are continuing to work on a permenant fix to deploy over the next hour and we will continue to keep you updated on the progress and inform you of any further downtimes required. In the mean time please use the dedicated search for every area, rather than Global Search (ie. Go to Projects and search for a project, Contacts to search a contact etc.) This will remain a top priority until it is completely resolved and we're incredibly grateful for your patience and understanding.
- identified Nov 07, 2022, 02:10 AM UTC
Our Engineering team will be deploying a fix, there will be a 5-10 minute downtime from 1:20 PM AEST.
- monitoring Nov 07, 2022, 03:11 AM UTC
Our Engineering team have applied a fix, this fix has resolved both the platform slowness and the Global Search and everything is running as expected again. The support team will continue to keep an eye out for any further reports of slowness, and our Engineering team will continue to monitor performance in the background.
- resolved Nov 07, 2022, 03:36 AM UTC
The fix we applied in our earlier update has resolved this incident. Global search is now functioning correctly and the loading speeds across Synergy are back to normal. Our deepest apologies once again for the disruption this caused you today, and a huge thank you for your patience while our teams worked to get this resolved.
- postmortem Nov 07, 2022, 04:44 AM UTC
Between the hours of 10:00 to 13:20 \(AEST\) on 07/11/2022 multiple users encountered slow performance when using Synergy Cloud The event was triggered by a change introduced as part of the Synergy November release which related to making global search more reliable. A complex retry issue caused degraded performance to customers using Synergy. The event was detected by multiple customers as well as our internal systems. A change has been tested and implemented to revert the previous change and was confirmed working around 13:20 \(AEST\) 07/11/2022.