Mural incident

Users are unable to sign In to MURAL

Critical Resolved View vendor source →

Mural experienced a critical incident on August 9, 2021 affecting Authentication and Realtime collaboration, lasting 2h 12m. The incident has been resolved; the full update timeline is below.

Started
Aug 09, 2021, 06:35 AM UTC
Resolved
Aug 09, 2021, 08:48 AM UTC
Duration
2h 12m
Detected by Pingoru
Aug 09, 2021, 06:35 AM UTC

Affected components

AuthenticationRealtime collaboration

Update timeline

  1. investigating Aug 09, 2021, 06:35 AM UTC

    Users are currently unable to sign in to MURAL. We know this is a major service disruption for everyone. We're investigating the issue and will restore regular service ASAP. Please check our status page for the most up-to-date info 👉 status.mural.co/

  2. identified Aug 09, 2021, 06:55 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Aug 09, 2021, 07:48 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Aug 09, 2021, 08:48 AM UTC

    This incident has been resolved.

  5. postmortem Sep 02, 2021, 09:29 PM UTC

    **What happened?** At 06:15 UTC on August 9, 2021, we experienced a spike in CPU usage on our primary database servers. A very large number of simultaneous operations in a single workspace, specifically joining a workspace and moving content between workspaces, generated write conflicts and resulted in the primary database server locking up. Users that were logged in at the time had their sessions terminated and no new login requests could be processed. **‌Details and corrective actions** We identified the cause of the incident and initiated a fail-over to a new primary database server at 06:44 UTC. This was completed at 07:35 UTC, at which point full service was restored. We immediately started investigating the root cause of the write conflicts and optimizing the workflows for joining workspaces to prevent this from impacting system availability again. ‌**Summary** The outage resulted in 1 hour and 20 minutes of downtime. No data from prior to the outage was lost during this time. ‌**What we’ve done to avoid this happening again** As an immediate action we implemented an optimization to the workflow for joining the impacted workspace. We are working towards applying this update to all workspaces in an upcoming release.