Munisense experienced a minor incident on April 30, 2024 affecting Sensor Data Stream and External Message Queue and 1 more component, lasting 21h 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 30, 2024, 04:45 PM UTC
After the issue from this morning where a fix was implemented we again see the same behavior, we are investigating te problem to see if we need to make additional fixes of if this is a different problem. As of now the sound data might again be delayed. Our sincere apologies, we will do everything to fix this as soon as possible and investigate the root issue that is causing this.
- investigating Apr 30, 2024, 07:16 PM UTC
We are still investigating this issue. We cannot provide a timepath yet. It seems that msnp devices are less impacted them mssp devices.
- investigating Apr 30, 2024, 09:11 PM UTC
Update 23:10 We do not have a fix for the issue. Data is still delayed (a few minutes to an hour in extreme cases). We continue to work on a solution for this issue.
- identified May 01, 2024, 05:51 AM UTC
We have determined that high load is causing our issues. We are currently working on getting a new database server installed to help spread the load. We will update as soon as more information is available.
- monitoring May 01, 2024, 09:32 AM UTC
Our new database server is having effect, we see reduced load on our systems and nearly all devices should report in realtime again. We will continue our efforts to reduce the load and monitor this closely. We will update as soon as there is more information. If you have any questions please contact [email protected]
- resolved May 01, 2024, 02:42 PM UTC
Since ~11:00 we have not seen any more disturbances in the data flow. The new database server is working perfectly. It is possible that your devices has a small bit of data loss, most likely between 30-04 19:00 and 01-05 11:00. We do not expect large gaps in the data. In some cases we might be able to manually restore this. Please contact [email protected] if you notice this and would like us to investigate. Again our apologies for the inconvenience caused, we will do a full root-cause analysis regarding this issue.