Potential Sign On Issues

Update timeline

1. investigating Apr 07, 2021, 02:09 PM UTC

   We're hearing reports that some users are having an authentication issue when they navigate from the switchboard page. We're currently investigating the issue.

2. investigating Apr 07, 2021, 03:04 PM UTC

   We're continuing to investigate the issue, and have narrowed the scope of the concern to the switchboard.

3. investigating Apr 07, 2021, 03:33 PM UTC

   We were concerned this was impacting a large number of users, however, this is isolated to a small number of users. We're continuing to investigate the root cause.

4. investigating Apr 07, 2021, 05:19 PM UTC

   We are continuing to investigate this issue. We believe it may have to do with a browser update that some of our customers rolled out and we are working to verify whether that is actually the case.

5. identified Apr 07, 2021, 06:47 PM UTC

   We have identified that the problem is only happening with a few select customers. We believe there may have been a browser upgrade or device change overnight that introduced the issue. We have reached out to IT leadership at the identified customers and are having the conversation directly to learn more in an attempt to isolate the problem. If you're not in an active conversation with our IT team, your organization has not been identified as being affected.

6. monitoring Apr 07, 2021, 07:43 PM UTC

   We believe we have identified the issue as a misconfigured downstream provider server (a CloudFront server on the East Coast). The server has been disabled, and we are observing error rates dropping, returning to normal bounds. We are continuing to monitor and test.

7. resolved Apr 07, 2021, 10:33 PM UTC

   This incident has been resolved. Our service provider removed the malfunctioning CDN node, and as a result customers who were affected are now being directed to a functioning node, and are able to continue working.