SchemeServe experienced a minor incident on September 5, 2022 affecting 🎩 SchemeServe, lasting 16h 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 05, 2022, 03:05 PM UTC
We are currently investigating reports from some clients of slow loading speeds.
- resolved Sep 06, 2022, 07:34 AM UTC
This incident has been resolved.
- postmortem Sep 06, 2022, 11:55 AM UTC
SchemeServe uses a session state to check whether a user is authenticated to access the platform. This session state is temporarily locked during requests to stop the same requests from being submitted multiple times, most requests will complete in milliseconds however can sometimes take multiple seconds to run if they are larger requests. The slowdown that was affecting a small subset of customers yesterday was caused when the session state for requests was being locked as expected, however, was not being released correctly at the end of the request. As a result, subsequent requests using the same logged-in user session were timing out as they could not validate whether the user was authenticated or not. This issue affected about 7% of SchemeServe logged-in users We are currently working to remove the session state block to make sure that this doesn’t happen in the future.