Total Synergy incident
27/09/2023 - Synergy App Performance Issues
Total Synergy experienced a major incident on September 26, 2023 affecting Synergy App, lasting 4h 13m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 26, 2023, 11:02 PM UTC
We're investigating multiple reports of platform slowness this morning, 27/09/2023 (9 AM AEST) Our engineering team are investigating this with high priority and we will provide updates as they become available. We're incredibly sorry for any disruption this is causing you today and endeavour to have this resolved as quickly as possible.
- identified Sep 26, 2023, 11:06 PM UTC
Our engineering team have identified the problem causing the current platform slowness in Synergy and will be applying a fix within the next few minutes. There will an expected downtime of 5-10 minutes from 9:05 AM AEST, we will continue to provide updates as they become available.
- monitoring Sep 26, 2023, 11:21 PM UTC
Our Engineering team have restarted the Synergy Database, and this has resolved the performance issues this morning (27/09/2023). The support team will continue to keep an eye out for any further reports of slowness, and our Engineering team will continue to monitor performance in the background. Thank you for your patience, and apologies again for the disruption this caused you today.
- resolved Sep 27, 2023, 03:15 AM UTC
This incident has been resolved and performance is back to normal.
- postmortem Sep 28, 2023, 06:34 AM UTC
Yesterday \(September 27, 2023\) at 08:00 - 09:20 AEST, one of the Synergy data components was experiencing a heavy load. This load was non-standard and unexpected, resulting in low performance on some of Synergy's operations. Following this discovery, our team reset the impacted component and increased its capacity. This was conducted with a downtime of less than 5 minutes.