Webshipper experienced a critical incident on February 9, 2022 affecting API and Frontend and 1 more component, lasting 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 09, 2022, 01:35 PM UTC
We are currently investigating this issue.
- resolved Feb 09, 2022, 01:54 PM UTC
This incident has been resolved.
- postmortem Feb 09, 2022, 02:04 PM UTC
At 2022-02-09 14:35 CET our primary database had a malfunction which caused an automatic failover. Usually automatic failovers stabilise in less than a minute with minimal effect on update, but because of an automatic cleanup job, our primary API instances were not able to respawn new instances since essential build assets had been cleaned up. Webshipper staff, after identifying the issue, instantiated a new deploy of the platform which added the essentials build assets. To mitigate this happening in the future we have modified the cleanup job to not remove essentials assets and will put effort into minimising the time needed to do a full re-deploy. We sincerely apologize and any inconvenience during this downtime. Best, The Webshipper Team