Mural incident

Access blocked when logging in to MURAL

Minor Resolved View vendor source →

Mural experienced a minor incident on November 24, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
Nov 24, 2022, 08:34 PM UTC
Resolved
Nov 24, 2022, 08:00 PM UTC
Duration
Detected by Pingoru
Nov 24, 2022, 08:34 PM UTC

Update timeline

  1. resolved Nov 24, 2022, 08:34 PM UTC

    At around 15:00 UTC we started to receive notifications from MURAL users that were unable to access their content. The issue appears to have been caused by one of MURAL's security measures, which identified certain actions as potentially malicious. This resulted in a number of MURAL users having their IP addresses blocked. We have modified this configuration to ensure users can log in. We are investigating the root cause and will publish a postmortem on our Status Page once we have full details and corrective actions.

  2. postmortem Dec 07, 2022, 12:51 PM UTC

    **What happened?** At around 16:15 UTC we started to receive notifications from Mural users that were unable to log in to Mural. At 17:01 the issue was escalated to our major incident team. Initial investigations indicated the issue coincided with an update to to the Mural web application earlier in the day. Following the update, a higher volume of web traffic was being generated. This triggered security alerts in our firewall and blocked Mural users from logging in. At 17:18 we adapted the web traffic thresholds in line with the new expected volume of traffic. This enabled Mural users to log in again and restored full service. ‌ **Summary** This incident impacted logins from 16:15 UTC to 17:18 UTC, for a total of 1 hour 3 minutes of service interruption. ‌ **What we've done to avoid this happening again** The previous firewall ruleset had a low tolerance for deviations from the expected volume of web traffic. We have updated the firewall ruleset to account for the increased web traffic and ensure there are no further service interruptions.