Users are unable to sign In to MURAL

Update timeline

1. investigating Sep 23, 2022, 10:09 AM UTC

   Users may be intermittently unable to sign in to MURAL. We know this is a major service disruption for everyone. We're investigating the issue and will restore regular service ASAP. Please check our status page for the most up-to-date info 👉 status.mural.co/

2. monitoring Sep 23, 2022, 10:28 AM UTC

   The issue has been mitigated and users are now able to sign back into MURAL; we continue to monitor the situation. Once again, we'd like to apologize for any disruption this might have caused. Please check our status page for the most up-to-date info 👉 status.mural.co/

3. resolved Sep 23, 2022, 11:31 AM UTC

   The issue is now resolved and users are able to sign back into MURAL correctly. Once again, we'd like to apologize for any disruption this might have caused.

4. postmortem Oct 13, 2022, 09:42 PM UTC

   **What Happened?** At 09:43 UTC on September 23rd 2022 our automated monitoring indicated that users were receiving intermittent errors when trying to access MURAL. We determined that the issues were being caused by a recent code change which unexpectedly impacted our infrastructure capacity. We were able to provisionally mitigate this by increasing capacity, resolving the issue definitively through an updated code change. ‌ **Summary** This incident impacted service availability from 09:43 UTC to 10:42 UTC, for a total of 1 hour of intermittent downtime. No data was lost, however some members would have experienced troubles accessing their MURAL data. ‌ **What we’ve done to avoid this happening again** We are documenting the constraints identified through this outage as guidance for future changes. In addition to this we are reviewing our automated infrastructure capacity management to ensure we stay within acceptable parameters. We are also reviewing protocols to maintain system stability, should this specific issue occur again.