ServiceMax Support Announcement - Salesforce Services Disruption

Update timeline

1. investigating Nov 15, 2024, 08:22 AM UTC

   Description of Issue: The Salesforce Technology team has became aware of a service disruption affecting multiple instances. During the disruption, customers can’t access the Salesforce services. Action Being Taken: Multiple Salesforce teams are actively engaged in investigating the cause and exploring multiple paths to resolution. You can monitor the current status here: https://status.salesforce.com/generalmessages/1476 By Whom: Salesforce Next Update: 10:00 UTC or earlier if there is a Salesforce update

2. investigating Nov 15, 2024, 08:59 AM UTC

   We are continuing to investigate this issue.

3. investigating Nov 15, 2024, 09:16 AM UTC

   At 05:55 UTC on November 15, 2024, the Salesforce Technology team became aware of a service disruption impacting instances under IT3, IT4, IA7, HN3, and HN4 Data Centers. At approximately 08:15 UTC, Salesforce teams began taking steps to control excessive traffic, network utilization, and database stability to mitigate the impact quickly. Salesforce don’t yet understand when customers may begin experiencing improvement as a result of these actions. Initially, this was posted as a Salesforce Informational Message, but as instances were identified, Salesforce have updated this to an Incident. Status updates can be found here - https://status.salesforce.com/incidents/13457 Update will be provided by 11:00am UTC or earlier

4. investigating Nov 15, 2024, 11:21 AM UTC

   Salesforce have successfully made a change on a sandbox instance, where they observed improved database connectivity performance. They are in the process of making this change on a small number of production instances and will analyze the results. Next Update: 12:00 UTC or earlier if there is a Salesforce update

5. identified Nov 15, 2024, 12:45 PM UTC

   Description of Issue: The Salesforce Technology team has became aware of a service disruption affecting multiple instances. During the disruption, customers can’t access the Salesforce services. Update: Salesforce continues to see positive results following the cchanges on a number of instances. The rollback is being done in a staggered manner due to the scale of the issue. As rollbacks complete, affected customers should begin to see improved performance. The rollback efforts are expected to take several hours. Salesforce updates can be found here - https://status.salesforce.com/incidents/13457 Next ServiceMax Update: 15:00 UTC or earlier if there is a Salesforce update

6. identified Nov 15, 2024, 02:10 PM UTC

   Description of Issue: The Salesforce Technology team has became aware of a service disruption affecting multiple instances. During the disruption, customers can’t access the Salesforce services. Update - The rollback is confirmed to successfully resolve the issue and the team will continue to implement the rollback across the remaining instances. The team has developed a script to perform this rollback on the impacted instances in groups to speed up the resolution. Next ServiceMax Update: 17:00 UTC or earlier if there is a Salesforce update

7. monitoring Nov 15, 2024, 04:35 PM UTC

   Description of Issue: The Salesforce Technology team has became aware of a service disruption affecting multiple instances. During the disruption, customers can’t access the Salesforce services. Update: Salesforce has rolled back all production instances and is conducting final checks. The team is now focusing on the sandbox environment in groups, which they expect to take considerable time. Next update: Saturday 15 November

8. resolved Nov 16, 2024, 12:15 PM UTC

   Resolved: After a period of monitoring, the Salesforce Technology team confirmed that Core services were restored as of 15:22 UTC on Friday 15 November. Salesforce will perform a full investigation of the incident, confirming the technical trigger, underlying cause, and preventive action to avoid a repeat in the future. Final update from Salesforce is here - https://status.salesforce.com/incidents/13457