Aptoma incident
Currently not possible to edit articles in DrPublish 5
Aptoma experienced a major incident on February 22, 2021 affecting DrPublish - GUI, lasting 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 22, 2021, 10:08 AM UTC
We have an incident with collaborative editing, which affects ability to write. We are working on a fix, and expect to report back soon.
- monitoring Feb 22, 2021, 10:11 AM UTC
We have corrected the situation and are monitoring.
- investigating Feb 22, 2021, 10:30 AM UTC
Problems persist. We are still investigating.
- monitoring Feb 22, 2021, 10:36 AM UTC
A fix has been implemented, and we are monitoring. NB: Please reload DrPublish before continuing to work as normal.
- resolved Feb 22, 2021, 10:42 AM UTC
Our understanding is that it is working for all users who have reloaded DrPublish. Please contact [email protected] immediately if you observe otherwise.
- postmortem Feb 22, 2021, 11:02 AM UTC
At 11:05 we deployed a misconfiguration to a new staging-environment for collaboration services in DrPublish, which affected production. The misconfiguration have been resolved for good, and we have made sure a similar effect from staging to production cannot happen again for this reason in the future. While issuing a fix at 11:10, we made a separate mistake and rolled back to a version with a known bug. This affected users until 11:34. At 11:34 the situation was permanently resolved, and all users were instructed to reload at 11:36. In the bigger picture, we see that our introduction of a new technology, while having solved many problems for users in the daily, has caused some procedural changes in our development and operations team, which have caused two separate incidents \(see also incident 12. February [https://www.aptomastatus.com/incidents/657h91dlfj9d](https://www.aptomastatus.com/incidents/657h91dlfj9d)\) We are, and will still be, working to establish robust development and operations workflows to ensure intermittent problems caused by us will not persist over time. We are sorry for the problems caused.