ServiceChannel Performance Degradation Remediation

Update timeline

1. monitoring Jun 19, 2023, 09:04 PM UTC

   In an effort to address ongoing performance issues during periods of heavy platform utilization, the ServiceChannel SRE and Application Engineering teams are preparing to roll our platform back to its 24 May 2023 revision. We expect to complete the rollback on 20 June by 8 am EDT. Platform changes implemented after that date will be unavailable until we are able to reintegrate them. This may take several days. Work orders, invoices, and other transactional activities added to the platform after 24 May 2023 WILL NOT be impacted and remain available. We regret the inconvenience and will keep you posted as we continue to address ongoing performance issues. Thank you for your continued understanding and patience.

2. resolved Jun 20, 2023, 04:29 PM UTC

   As a part of our ongoing investigation, the ServiceChannel SRE and Application Engineering teams have explored a range of scenarios that may have contributed to our current performance issues during periods of high utilization. New information has come to light that indicates the problem could be related to aspects beyond the software itself, which has prompted us to reevaluate the need for the rollback and consider more targeted solutions. We made the decision to postpone the previously-planned rollback while we conduct a thorough examination of this new information. We believe that this decision is in the best interest of maintaining the optimal performance of the system and minimizing any disruptions to your experience. As always, we appreciate your understanding and patience.

3. postmortem Jul 11, 2023, 03:20 PM UTC

   Please see the general postmortem at [https://status.servicechannel.com/incidents/cvp26brsbwl8](https://status.servicechannel.com/incidents/cvp26brsbwl8) for a comprehensive description of work to remediate platform performance issues.