Golive Cloud - Major system outage

Update timeline

1. identified Oct 08, 2020, 05:04 PM UTC

   The root cause has been identified, we are working on the problem resolution.

2. identified Oct 08, 2020, 05:11 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Oct 08, 2020, 06:23 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Oct 08, 2020, 07:05 PM UTC

   A fix has been implemented and the service is up, except email notifications. We are now monitoring the platform.

5. resolved Oct 09, 2020, 01:39 AM UTC

   All Golive Cloud services are now back to normal.

6. postmortem Oct 09, 2020, 05:40 AM UTC

   We apologize for inconvenience caused by this first major outage of Apwide Golive Cloud. **Root cause** Outage was caused by a failure of the middleware managed by our hosting provider. **What we have done to restore the service** We have first safely transferred all customer data to an alternate data center. We have then deployed our applicative stack to fully restore the service for our customers. **What we have learned from this incident** * low level infrastructure or middleware failures happen an may happen again in the future * monitoring of our services works well. We were instantly aware of the incident * we are able to rebuild from scratch our productive infrastructure. This means that our disaster recovery procedure \(DRP\) is fully operational **What we will improve for the future** * we will improve our DRP in order to reduce the outage duration if we have to switch again from a data center to another * we will better integrate Status Page to improve communication about status of our services with our customers ‌ Thanks for having read this postmortem and for trusting Apwide Golive. We are at your disposal to answer to your [questions](https://jira.apwide.com/servicedesk). ‌ Enjoy your day, Kind Regards, ‌ Guillaume Vial / David Berclaz CEO’s