Mural experienced a minor incident on July 14, 2023 affecting Authentication and Learning and 1 more component, lasting 2h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 14, 2023, 11:26 AM UTC
Some users are unable to log in to Mural via SSO, with an error message of “invalid signature”. We are investigating and will restore regular service as soon as possible.
- investigating Jul 14, 2023, 12:24 PM UTC
We are continuing to investigate this issue.
- monitoring Jul 14, 2023, 12:59 PM UTC
We've implemented a fix that should resolve the issue. We're monitoring the results to make sure this has the intended effect and will update this message as soon as we've been able to confirm this.
- resolved Jul 14, 2023, 01:56 PM UTC
The fix we implemented earlier has been successful in resolving the issue and full service has been resolved. We apologize for the inconvenience this interruption caused, and will be conducting a full review to avoid a repeat in the future.
- postmortem Jul 18, 2023, 03:40 PM UTC
**Summary** On 13th July 2023, we implemented a change to our SAML SSO login flow. This change was released to production at 7pm UTC and we started receiving sporadic reports of login issues that were treated as isolated incidents, as the vast majority of Mural users were able to continue accessing without issues. By 11am UTC on the 14th July, the volume of login issues had increased and we activated our taskforce to investigate the cause. By 11:25 UTC we identified a potential cause of the issue and started preparing a fix. At 11:45 UTC we made the decision to roll back the change from the previous day, which was completed at 12:15 UTC. As of 12:15 UTC login behavior returned to normal parameters. **What we've done to avoid this happening again** The change we implemented on Thursday afternoon was rolled back and we are preparing a new change that will avoid impacting the SAML SSO login flow.