Total Synergy incident

Login Page - Request blocked when logging in

Major Resolved View vendor source →

Total Synergy experienced a major incident on April 13, 2025 affecting Synergy App, lasting 2h 36m. The incident has been resolved; the full update timeline is below.

Started
Apr 13, 2025, 11:01 PM UTC
Resolved
Apr 14, 2025, 01:37 AM UTC
Duration
2h 36m
Detected by Pingoru
Apr 13, 2025, 11:01 PM UTC

Affected components

Synergy App

Update timeline

  1. investigating Apr 13, 2025, 11:01 PM UTC

    We're currently investigating multiple reports of users being unable to login this morning, 14 April, 2025. Our development team are investigating this with high priority and we will provide updates as they become available. This is only impacting Username & Password log-ins. Social log-ins (MS365 etc) are not impacted so please log in using alternative options where possible. We apologies any disruption this is causing and we endeavour to have this resolved as quickly as possible.

  2. identified Apr 13, 2025, 11:34 PM UTC

    Our development team have identified the problem causing the current login issues Total Synergy. There will be downtime of up to 10 minutes from 9:35 AM AEST, we will continue to provide updates as they become available.

  3. resolved Apr 14, 2025, 01:37 AM UTC

    Between approximately 8:00 AM and 9:40 AM AEST on 14 April 2025, we experienced a login outage that prevented some users from accessing Total Synergy. Our development team has implemented a fix, and users are now able to log in successfully. A permanent hotfix will be released in the coming days to prevent this issue from reoccurring. We sincerely apologise for the disruption and appreciate your patience as we resolved the issue.

  4. postmortem Apr 22, 2025, 12:19 AM UTC

    **Issue Summary:** Between **8:37 AM and 9:35 AM AEST on April 14, 2025**, a platform change caused our system to apply login rate limits based on a shared gateway IP address, rather than individual user IPs. As a result, many valid login attempts were blocked once the shared threshold was reached. Approximately **70% of users logging in via email/username and password** were affected during this time. It did not impact users logging in via Single Sign-On\(SSO\) and social logins. **Resolution:** * At **9:34 AM AEST on April 14**, we temporarily increased the login rate limit to immediately restore access. * A **permanent fix** was deployed at **5:00 AM AEST on April 15,** which now correctly identifies and applies rate limits based on individual user IPs. The issue has been fully resolved. We are also implementing additional safeguards to help prevent similar issues in the future**.**