BEE Plugin incident

Intermittent authorization issues

Notice Resolved View vendor source →

BEE Plugin experienced a notice incident on December 18, 2017, lasting 1d 10h. The incident has been resolved; the full update timeline is below.

Started
Dec 18, 2017, 09:33 PM UTC
Resolved
Dec 20, 2017, 08:22 AM UTC
Duration
1d 10h
Detected by Pingoru
Dec 18, 2017, 09:33 PM UTC

Update timeline

  1. investigating Dec 18, 2017, 09:33 PM UTC

    We received reports of intermittent authorization issues related to the UID validation. The editor loads normally when the page is refreshed.

  2. monitoring Dec 19, 2017, 10:04 AM UTC

    The issue was identified as a problem with AWS instances that had been replaced. Sice that, the loading issue stopped in our tests.

  3. resolved Dec 20, 2017, 08:22 AM UTC

    We found that one of the load-balanced machines in our AWS environment had become corrupted, which led to the intermittent issues. The AWS environment has been updated and the issue appears to have been resolved. In the short term, we implemented a few changes to minimize any chances of recurrence of this problem: - We increased the authorization token life to 120 seconds. - We modified the UID validator endpoint so that it will trigger a retry request if authorization fails. In the medium term, we plan to review the entire authorization system to make it even more robust and scalable.