BEE Plugin experienced a major incident on September 25, 2024 affecting API, lasting 22h 42m. The incident has been resolved; the full update timeline is below.
Affected components
API
Update timeline
- investigating Sep 25, 2024, 03:47 PM UTC
An error occurred with our co-editing worker Healthcheck starting 2024-09-25 15:09:35.656UTC
- monitoring Sep 25, 2024, 03:49 PM UTC
A fix has been implemented at 2024-09-25 15:27:05.790 UTC and we are monitoring the results
- resolved Sep 26, 2024, 02:30 PM UTC
The co-edit worker services were redeployed, effectively recycling old connections and allowing new services to establish connections. The number of co-edit workers was also reduced to slow the rate of connection accumulation. An action plan and mitigation strategy were implemented to prevent reoccurrence.