Presslabs experienced a major incident on November 12, 2020 affecting Europe and US and 1 more component, lasting 1h 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 12, 2020, 08:30 PM UTC
A set of cascading events led to backends being restarted. This in turn puts a heavy load on the git service. We are working on reducing the load on the git service so that the backends can be restarted.
- identified Nov 12, 2020, 08:37 PM UTC
The source of these issues was a latent bug, which quickly scaled down to 0 than back up the backends. We've reverted the faulty deploy and we are waiting for the sites to recover.
- monitoring Nov 12, 2020, 08:54 PM UTC
The sites are recovering and we are currently prioritizing live sites so dev sites might take some time to recover. Thank for your patience!
- monitoring Nov 12, 2020, 08:55 PM UTC
We are continuing to monitor for any further issues.
- monitoring Nov 12, 2020, 09:43 PM UTC
All sites are back up. We are continuing to monitor the situation.
- resolved Nov 12, 2020, 10:24 PM UTC
This incident has been resolved.