BenchPrep experienced a notice incident on January 3, 2022 affecting API and Learning Application, lasting 4h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 03, 2022, 08:30 PM UTC
We are currently investigating login issues. We will keep posted on updates.
- identified Jan 03, 2022, 08:47 PM UTC
The issue is identified and we are in the process of fixing it.
- identified Jan 03, 2022, 09:22 PM UTC
We restarted our Redis instance and traffic moved to the Dallas data center. Still working on fix.
- identified Jan 03, 2022, 11:18 PM UTC
We have been running on Dallas past few hours without any issues and performing full Redis replica restoration currently.
- identified Jan 04, 2022, 12:03 AM UTC
Redis replication restores completed and failover the Redis services to different node now.
- monitoring Jan 04, 2022, 12:28 AM UTC
We were able to successfully failover the Redis nodes and everything looks good, Now traffic serves to both data centers and monitoring the applications. We will keep monitoring and will update the final status.
- resolved Jan 04, 2022, 01:04 AM UTC
Redis fix is applied and the issue is resolved now. We continue to monitor the application.