Sympa HR experienced a minor incident on November 22, 2024 affecting System availability, lasting 7d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 22, 2024, 07:02 AM UTC
We are currently experiencing some errors and slowness across the system. Our team is actively investigating the issue to identify the root cause and implement a resolution. We will provide updates as soon as we have more information. Thank you for your patience and understanding.
- identified Nov 22, 2024, 02:05 PM UTC
We have identified the root cause of the errors and slowness in the system. A fix involving optimizing the database has been scheduled for implementation this evening to minimize disruption. Thank you for your continued patience and understanding.
- identified Nov 25, 2024, 07:35 AM UTC
We are continuing to experience errors and slowness across the system. Our team is actively working to resolve the issue and is now in communication with Microsoft for support. We are exploring potential solutions with Microsoft and will provide updates as soon as we have more information. Thank you for your patience and understanding.
- identified Nov 25, 2024, 05:09 PM UTC
We are continuing to work on a fix for this issue.
- identified Nov 25, 2024, 05:09 PM UTC
We are currently addressing an issue impacting the issue of our system. As part of the resolution, a small system update will be performed shortly. This may cause minor slowness temporarily, but it is expected to improve the overall situation. We appreciate your patience and understanding as we work to enhance system stability.
- monitoring Nov 25, 2024, 07:51 PM UTC
A fix has been implemented and we are monitoring the results.
- identified Nov 26, 2024, 08:50 AM UTC
We are aware that the system is still experiencing slowness. Our team is actively working to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience. Further updates will be provided as soon as we have more information.
- identified Nov 26, 2024, 06:05 PM UTC
We are going to address couple of issues impacting our system. As part of the resolution, couple of small system updates will be performed during this evening. This may cause minor slowness temporarily, but it is expected to improve the overall situation. We appreciate your patience and understanding as we work to enhance the system.
- monitoring Nov 26, 2024, 09:00 PM UTC
A fix has been implemented and we are monitoring the results.
- identified Nov 27, 2024, 07:17 AM UTC
We are aware that the system is unfortunately still experiencing some slowness and issues. Situation has improved, but we are not yet happy for the situation. Our team is actively working to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience. Further updates will be provided as soon as we have more information.
- identified Nov 27, 2024, 02:12 PM UTC
We have identified the root cause of the slowness as being related to Azure. Our team is actively collaborating with Microsoft Azure support to address the issue and restore optimal performance as soon as possible. We will provide further updates as we make progress. Thank you for your patience and understanding.
- identified Nov 28, 2024, 08:03 AM UTC
We are continuing to work on the improvements on multiple tracks, including heavy collaboration with Microsoft Azure support team. We are closely monitoring system performance and working on incremental enhancements to be deployed during today! Thank you for your continued patience and understanding.
- identified Nov 28, 2024, 06:02 PM UTC
We are going to address couple of issues impacting our system. This includes moving some of the critical services on new Azure resources trying to improve the situation despite the Azure issues faced. As part of the resolution, couple of small system updates will be performed during this evening. This may cause minor slowness temporarily, but it is expected to improve the overall situation. We appreciate your patience and understanding as we work to enhance the system.
- monitoring Nov 28, 2024, 09:51 PM UTC
We have now deployed multiple fixes and optimizations to the system, and moved multiple services to different, more robust Azure resources. Our team continues to actively 1) collaborate with Microsoft Azure support, 2) optimize which Azure resources we have in use and 3) apply modifications to the software to be resilient from existing Azure issues. We are closely monitoring the situation, and will provide further updates as we make progress. Thank you for your patience and understanding.
- monitoring Nov 29, 2024, 07:28 AM UTC
Yesterday night changes and optimizations have had very positive effect. We continue working on the futher optimizations and are monitoring the system very closely.
- resolved Nov 29, 2024, 08:37 AM UTC
The incident has been resolved. Everything looks good, back to normal and actually to even better state.
- postmortem Nov 29, 2024, 02:16 PM UTC
Our services experienced intermittent slowness due to Azure conducting extended maintenance on their infrastructure. To resolve the issue, we migrated our critical services to more robust resources and made several modifications to enhance their performance and reliability. We apologize for the inconvenience caused and appreciate your understanding as we continue to improve our services.