360Learning experienced a notice incident on September 19, 2023 affecting Web Application and API, lasting 1h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 19, 2023, 02:28 PM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- investigating Sep 19, 2023, 03:07 PM UTC
Service unavailable. An issue has been detected on our servers. The service is currently down. We're investigating the root cause and working on a solution. We will send an additional update around 5:30 p.m. CET.
- monitoring Sep 19, 2023, 03:32 PM UTC
A fix has been deployed to production. We will keep monitoring and take further action if needed.
- resolved Sep 19, 2023, 03:58 PM UTC
The issue has been resolved. We apologize for any inconvenience caused by this incident
- postmortem Sep 21, 2023, 04:45 PM UTC
## Incident report **Service slow or unavailable on Sept 19, 2023, for 1 h 29 minutes** Between the hour of 16:02 to 17:31 CET on September 19st, 2023, our users encountered a slow or unresponsive web application and mobile application, as well as issues on 360Learning Public API. The event was triggered by a routine upgrade operation on the main database, that unfortunately placed the database into an unstable state. Our teams detected the incident immediately due to our monitoring system. We decided to remove the faulty database nodes from operation, and to restart them. Around 17:02 CET the service was back for most users, but slownesses remained until 17:31 CET. Following this incident, we deployed further fixes on the next days, causing short episodes of a slower than usual service. At the date of writing, the incident is now completely resolved. We have taken preventive measures to minimize the impact of this type of incident in the future. We apologize for the inconvenience caused by the incident.