Mural incident

Customers on private tenants with BYOK are not able to log in

Minor Resolved View vendor source →

Mural experienced a minor incident on May 2, 2022 affecting Authentication and Billing and 1 more component, lasting 2h 25m. The incident has been resolved; the full update timeline is below.

Started
May 02, 2022, 01:58 PM UTC
Resolved
May 02, 2022, 04:24 PM UTC
Duration
2h 25m
Detected by Pingoru
May 02, 2022, 01:58 PM UTC

Affected components

AuthenticationBillingLearningCanvasWebsiteExportsIntegrationsMural DatabaseNotificationsRealtime collaboration

Update timeline

  1. investigating May 02, 2022, 01:58 PM UTC

    Some users are currently unable to sign in to MURAL. We know this is a major service disruption. We're investigating the issue and working to restore regular service as soon as possible. Please check our status page for the most up-to-date info status.mural.co

  2. monitoring May 02, 2022, 02:17 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring May 02, 2022, 03:05 PM UTC

    We are continuing to monitor for any further issues.

  4. resolved May 02, 2022, 04:24 PM UTC

    This incident has been resolved

  5. postmortem May 17, 2022, 04:56 PM UTC

    **What happened?** A small number of customers using BYOK experienced an issue with logging in to the MURAL application. Authentication requests were being redirected from their customer tenants to the main [app.mural.co](http://app.mural.co) tenant, resulting in their users being unable to log in. Initial reports of this were received from MURAL customers on Sunday May 1st. Further reports were received on Monday May 2nd, at which point the issue was classified as a major incident and our task force was engaged. ‌ **Summary** MURAL’s task force started investigating at 13:58 UTC on Monday May 2nd. The cause of the issue was identified as being an incorrect configuration in how users on custom tenants were being routed through the login process. Once the issue was identified, a fix was rolled out at 14:07 UTC. This immediately corrected the login flow for clients using BYOK. ‌ ‌**What we've done to avoid this happening again** Parts of the login flow for BYOK clients had been hard-coded. We have changed this to be dynamically generated, reducing the risk of code-based errors. We are also researching avenues to improve the automated monitoring of login issues.