Total Synergy experienced a critical incident on December 19, 2022 affecting Synergy App, lasting 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 19, 2022, 05:43 AM UTC
At 4:30 PM AEDST, Monday 19 December, 2022, our internal systems notified us of an issue with Synergy Cloud being unresponsive, and that some customers are unable to log in.. At 4:40 PM AEDT, we deployed a change to the environment, and are currently assessing if this change has been successful in solving the issue. We are incredibly sorry for any inconvenience this may be causing you, but please rest assured we are working on having this resolved as quickly as possible. We appreciate your patience and understanding.
- monitoring Dec 19, 2022, 06:05 AM UTC
In an effort to improve the current situation, we have increased the capacity of our environment and are seeing significant improvement in loading times. We'll continue to actively monitor the performance in the short term, and provide further updates before 5:30 PM AEDT.
- resolved Dec 19, 2022, 06:42 AM UTC
The change we deployed at 4:40 PM AEDT has resolved the issue and Synergy Cloud is operational. We're incredibly appreciative of the patience during this performance degradation, and once again we are so sorry for any and all disruption to your daily workflow this may have caused.
- postmortem Dec 21, 2022, 12:19 AM UTC
On 19/12/2022 at 16:25-16:50 AEDT there was excessive load on the production environment which was mitigated by scaling a specific component in production which was reaching its scale limits. The development team will keep monitoring this component to establish whether this was a one off end of year event or a different issue.