Induction Status Broken (now resolved)

Update timeline

1. investigating Sep 26, 2022, 11:21 PM UTC

   We are aware of an issue causing induction statuses to show incorrectly in the web portal. This issue is actively being investigated, and we will release a fix as soon as we can. We believe the the issue is only affecting viewing the induction status of individuals with induction workflows (submitting, reviewing, approving) being unaffected and working as normal. Any data warehouse dashboards (PowerBI, DataStudio) and the customer API should be unaffected. The issue was most likely introduced in our latest release of an internal API (not our customer API) last night whilst updating an endpoint. The update was to fix a performance problem causing timeouts on the attendance register, but the fix seems to have interacted unexpectedly with the inductions system. This is a working hypothesis and the cause hasn't yet been isolated and confirmed. Updates will be posted on our statuspage (https://signonsite.statuspage.io/) as we have them. The issue is expected to be resolved this morning.

2. investigating Sep 26, 2022, 11:22 PM UTC

   We are continuing to investigate this issue.

3. investigating Sep 26, 2022, 11:36 PM UTC

   We are continuing to investigate this issue.

4. identified Sep 26, 2022, 11:45 PM UTC

   The cause of the issue has been confirmed to be the change we made late yesterday to our internal attendances endpoint (Confirming our hypothesis). A fix is being worked on now and should be released into production within the next 2 hours.

5. monitoring Sep 27, 2022, 01:07 AM UTC

   We have released a fix & all systems should be back to normal. We will be monitoring support & metrics for a period to ensure the fix resolves the issue. If you continue to experience issues, please contact [email protected] as soon as possible.

6. resolved Sep 27, 2022, 02:26 AM UTC

   This incident has been resolved.