Sense experienced a critical incident on September 6, 2024 affecting Note Writebacks, lasting 3h 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 06, 2024, 04:55 PM UTC
We have received reports that Messaging Note Writebacks are not recording into the respective ATS records. Our team is actively investigating the cause and working to have any missed messages backfilled as soon as possible
- identified Sep 06, 2024, 05:20 PM UTC
After review, our team was able to identify the root cause on the AWS server and have started to reprocess all missed writebacks. We have confirmed no notes will be missed during this delay and all notes should populate over the next few hours. We will ensure the note writeback queue is running as expected, and will update as we are seeing the flow stabilize once more
- monitoring Sep 06, 2024, 06:43 PM UTC
We are continuing to see the backfill of any missed writebacks decrease, and the queue stabilize to allow any new writebacks to populate in the expected timeframe once more. We will continue to monitor to ensure we don’t see the queue increase once more, but do expect to be fully caught up within the next hour and a half. We will provide an update once we are seeing this caught up, or if we see this recur.
- resolved Sep 06, 2024, 08:00 PM UTC
After monitoring, our team has confirmed the note writeback queue has stabilized and we’re seeing all notes populate in the expected time once more. If you believe you are still experiencing delays, please reach out to our support team to submit a ticket at https://support.sensehq.co/hc/en-us/requests/new and we will review on a case by case basis