SA Forms cannot be found or cannot be started

Update timeline

1. investigating Apr 02, 2024, 06:55 AM UTC

   We are currently experiencing difficulties with delegated forms within our service automation module. Users may encounter issues with opening or submitting delegated forms. Our team is actively investigating the issue to identify the root cause and implement a resolution as soon as possible. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this issue promptly.

2. monitoring Apr 02, 2024, 07:15 AM UTC

   We have identified the issue and implemented a solution. All services should have been restored. We are currently monitoring the situation.

3. resolved Apr 03, 2024, 06:33 AM UTC

   This incident has been resolved.

4. postmortem Apr 03, 2024, 06:40 AM UTC

   Due to a misconfiguration, a significant number of messages in our system were incorrectly marked as errors. These erroneously flagged messages did not impact functionality for our customers. However, what did affect our customers was that the system became overwhelmed on Monday evening due to the high volume of errors, which couldn't be processed. As a result of this overload, other messages couldn't be handled, putting the system under heavy strain. It took some time to recover from this, and during that period, customers experienced slower system performance. To prevent similar issues in the future, we have tightened our monitoring procedures and identified a bug that we will address in the upcoming release.