Socio incident
Degraded Performance with Video Rooms and Images on the Socio Platform
Socio experienced a major incident on December 7, 2021 affecting Platform and Web App, lasting 1d 10h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 07, 2021, 04:53 PM UTC
We’re currently experiencing degraded performance with the Socio Platform due to an ongoing issue with Amazon Web Services. Our Engineering team is working to identify the root cause and implement a solution. Images and videos on the Socio Platform may load slowly or not at all. Video Rooms are not working at this time. We'll provide an update within the next hour. Thank you for your patience as we work on a resolution.
- identified Dec 07, 2021, 05:50 PM UTC
Amazon Web Services, a Socio infrastructure partner, is currently reporting increased error rates and degraded service. This is directly impacting the Video Rooms and Socio Streaming. We're in contact with AWS and are carefully monitoring the issue. To receive alerts on the latest status, subscribe at https://status.aws.amazon.com/
- identified Dec 07, 2021, 08:15 PM UTC
We have confirmed with AWS that the issue is on their end. While we are looking at workarounds, we do encourage you to check the Amazon Web Services status page in the meantime. You can find that here - https://status.aws.amazon.com/ Thank you for your patience as we continue to work with AWS to provide a solution. We will send out another update within the hour.
- identified Dec 07, 2021, 09:13 PM UTC
AWS continues to work toward mitigation and are actively working on a number of different resolution actions. At this time, there is no ETA for a full recovery. We will continue to monitor the progress on our end and update when we have a better idea on a resolution. We will send out an update before 6:00 PM EST.
- identified Dec 07, 2021, 10:58 PM UTC
AWS have executed a mitigation which is showing significant recovery on their servers. They continue to closely monitor the health of the network devices and expect to continue to make progress towards full recovery. There is still no ETA for full recovery at this time. Please continue to watch https://status.aws.amazon.com/ for any updates from their side. We will send out an update before 10:00 PM EST.
- identified Dec 08, 2021, 03:00 AM UTC
Amazon has resolved its Network Device issue. While we believe that the underlying issue has been mitigated, we are still working on a full recovery across all systems. We will continue to prove updates as we work with AWS to restore all of our services. We will provide an update in the next two hours. Thank you for your continued patience as we work with our vendor to resolve this issue completely.
- monitoring Dec 08, 2021, 05:52 AM UTC
Amazon has mitigated the issues impacting the Socio Platform and continues to work for a full resolution. Our team has tested thoroughly and believes that the system should be working as expected. If you find this not to be the case, please notify us at [email protected] or chat in to our Support Team. We will continue to monitor the situation and provide an update before 8:30 AM EST, 08 DEC 2021. Thank you again for your patience as we work to provide a resolution to this situation.
- monitoring Dec 08, 2021, 01:10 PM UTC
Amazon has indicated that all services have returned to normal after the degraded performance in our system. Our team will continue to monitor the situation for the next 12 hours to make sure all services return to both stable and normal operation. Please do not hesitate to reach out to us at [email protected] or our live chat if you find that the platform is still having issues with performance degradation.
- resolved Dec 16, 2021, 03:41 AM UTC
This incident has been resolved.