Munisense incident

Noise portals unavailable and no new data available for water customers

Major Resolved View vendor source →

Munisense experienced a major incident on July 29, 2023 affecting Sensor Data Stream and Sensor Data Stream and 1 more component, lasting 14h 44m. The incident has been resolved; the full update timeline is below.

Started
Jul 29, 2023, 02:16 PM UTC
Resolved
Jul 30, 2023, 05:00 AM UTC
Duration
14h 44m
Detected by Pingoru
Jul 29, 2023, 02:16 PM UTC

Affected components

Sensor Data StreamSensor Data StreamCustomer PortalNotificationsPortal Webservice API

Update timeline

  1. investigating Jul 29, 2023, 02:16 PM UTC

    We are currently experiencing an outage for our noise measurement portals. We are currently investigating the issue. For our water customers: the portals are running but new measurement data might not be received currently.

  2. investigating Jul 29, 2023, 02:39 PM UTC

    An engineer has been dispatched to the datacenter to investigate the issue. We expect more information in about 60~90 minutes. As soon as we have more information this page will be updated. We apologize for any inconvenience. If you require assistance or want to contact us about the outage please email to: [email protected]

  3. identified Jul 29, 2023, 03:42 PM UTC

    Our engineer has determined that some hardware (a router) is defective and needs to be replaced. He brought a spare one and is currently working on that. A new update is expected within an hour.

  4. identified Jul 29, 2023, 04:29 PM UTC

    We are still working on configuring the replacement hardware. A new update will come as soon as there is more info.

  5. identified Jul 29, 2023, 05:01 PM UTC

    We are continue to work on restoring service as quickly as possible. Currently the portals may be available however not all data is being processed. We will update as soon as there is more information available.

  6. identified Jul 29, 2023, 05:49 PM UTC

    Currently data for our water customers is coming in again. If you are missing data please wait 24 hours to allow the device to try it again. For our noise customers: we are still working on resolving the last few of our systems.

  7. identified Jul 29, 2023, 06:24 PM UTC

    For most sensors connectivity should be restored, we are working to restore service for the last ones (connecting via ZGD gateway)

  8. monitoring Jul 29, 2023, 06:59 PM UTC

    The new router has been installed and configured and connectivity is restored. Most devices should have sent their delayed data by now with the exception of: - audio stream (streamed to the platform, data stored by nodes themselves should be ok) - any meter connected to a ZGD gateway (small white gateway), mostly mssp8 devices If you want to know any more information of have any questions about this outage please contact [email protected] We will monitor this for the next couple of hours before we set this page to resolved.

  9. identified Jul 29, 2023, 08:23 PM UTC

    An (unrelated) crash of the database server is breaking connectivity again. We are working to resolve this as soon as possible.

  10. identified Jul 29, 2023, 08:31 PM UTC

    Connectivity to the portals is restored, however one of our database servers is currently still unavailable so you might be unable to see (part of) the data.

  11. monitoring Jul 30, 2023, 12:40 AM UTC

    Database server is operational. We are monitoring the situation. We expect to change the issue to resolved this morning if we see no errors the upcoming night.

  12. resolved Jul 30, 2023, 05:00 AM UTC

    All systems are working correctly. The spare router will be replaced in the upcoming week.