Mural incident

Users experience icons disappearing on the mural and are no longer available in the icon library when searched

Major Resolved View vendor source →

Mural experienced a major incident on November 19, 2021 affecting Realtime collaboration, lasting 7h 44m. The incident has been resolved; the full update timeline is below.

Started
Nov 19, 2021, 09:40 AM UTC
Resolved
Nov 19, 2021, 05:25 PM UTC
Duration
7h 44m
Detected by Pingoru
Nov 19, 2021, 09:40 AM UTC

Affected components

Realtime collaboration

Update timeline

  1. investigating Nov 19, 2021, 09:40 AM UTC

    We're experiencing a service disruption causing icons to disappear on the mural and are no longer available on the icon library when searched. We're investigating the issue and will restore regular service ASAP.

  2. identified Nov 19, 2021, 10:16 AM UTC

    The cause of the issue has been identified and our team is working on fixing it.

  3. identified Nov 19, 2021, 12:06 PM UTC

    Our team continues to work on fixing the issue. We will post an update as soon as we have more information or in an hour’s time. We appreciate your patience and apologize for this inconvenience.

  4. identified Nov 19, 2021, 01:06 PM UTC

    Our team continues to work on this issue. We will post an update as soon as we have more information or in an hour’s time.

  5. identified Nov 19, 2021, 02:07 PM UTC

    Our team is still working on this issue. We will post an update as soon as we have more information or in an hour’s time.

  6. identified Nov 19, 2021, 03:10 PM UTC

    We are continuing to work on a fix for this issue. We will post an update as soon as we have more information or in an hour’s time.

  7. identified Nov 19, 2021, 04:11 PM UTC

    Our team continues to work on this issue. We will post an update as soon as we have more information or in an hour’s time. We appreciate your patience.

  8. monitoring Nov 19, 2021, 04:26 PM UTC

    We've implemented a fix that should resolve the issue with icons. We're monitoring the results to make sure this has the intended effect and will update this page within the next hour to confirm.

  9. resolved Nov 19, 2021, 05:25 PM UTC

    The fix we implemented has been successful in resolving the issue with icons and full service has been resolved. We apologize for the inconvenience this interruption caused and will be conducting a full review to avoid a repeat in the future.

  10. postmortem Nov 24, 2021, 04:03 PM UTC

    **What happened?** At 19:00 UTC on November 18 2021 we started to receive a small number of notifications of some [Noun Project icons](https://support.mural.co/en/articles/2113734-add-icons) disappearing from murals and not being visible in the search bar. During our initial investigations we were unable to reproduce the issue. We continued to receive reports and at 09:40 UTC on November 19 2021 we classified this as a critical incident. The relevant code for displaying icons was retained and no data was lost, however the icons were not being displayed. ‌ **Details and corrective actions** At 16:26 UTC we updated the configuration for the Noun Project integration, which restored full service with no data loss. ‌ **Summary** This incident resulted in a partial outage for a core service, lasting 7 hours and 44 minutes. MURAL remained operational throughout this incident and members were able to log in and access their data; however, some Noun Project icons were not visible in the search menu or on murals. ‌ **What we've done to avoid this happening again** We have resolved the configuration issue and restored full service with the use of icons. We are working on several short- and medium-term initiatives to improve the reliability of this service going forward.