IT Portal incident

Slow Performance in the US Cloud

Minor Resolved View vendor source →

IT Portal experienced a minor incident on August 14, 2024 affecting US Cloud, lasting 7d 1h. The incident has been resolved; the full update timeline is below.

Started
Aug 14, 2024, 05:24 PM UTC
Resolved
Aug 21, 2024, 07:18 PM UTC
Duration
7d 1h
Detected by Pingoru
Aug 14, 2024, 05:24 PM UTC

Affected components

US Cloud

Update timeline

  1. identified Aug 14, 2024, 05:24 PM UTC

    The issue has been identified and a fix is being implemented.

  2. identified Aug 15, 2024, 11:10 AM UTC

    We are still reviewing the slow performance seen by some cloud users. We have identified the slowness to one particular database server and are working with SQL Consultants to analyze this further.

  3. identified Aug 15, 2024, 06:05 PM UTC

    We are still working to determine the cause of the issue. We have opted to move half of the affected partners to a new SQL instance, and this will be done later tonight. You may experience a brief disconnect as we move databases.

  4. identified Aug 16, 2024, 11:43 AM UTC

    We thank you for your patience as we resolve this slowness issue for some partners. We plan on doing some more database changes tonight at 8PM. Thanks again.

  5. identified Aug 19, 2024, 01:25 AM UTC

    Databases are being moved to new SQL Servers

  6. monitoring Aug 19, 2024, 03:07 AM UTC

    Three new SQL Servers have been deployed and database migration has been completed. We will now monitor to see if performance is better.

  7. resolved Aug 21, 2024, 07:18 PM UTC

    After a few days of monitoring performance of the new SQL Instances, we have not noticed any performance issues. Please do reach out to support if you continue to experience delays.