Sympa HR Outage History

Started Nov 06, 2025, 08:51 AM UTC· resolved Nov 06, 2025, 06:11 PM UTC

1. identified Nov 06, 2025, 08:51 AM UTC

   We’ve identified an issue with a recent change to our access control logic. Some users can log in successfully but are unable to view data or resources they normally have access to. Our team is actively implementing a fix to restore full visibility and access as soon as possible. Your Sympa Team

2. identified Nov 06, 2025, 12:57 PM UTC

   We’re continuing work on the fix for the access control issue. Final testing is underway before deployment, which may take a few more hours. The fix requires careful validation to ensure a safe and complete resolution. Thank you for your patience and understanding while we complete this work.

3. monitoring Nov 06, 2025, 04:38 PM UTC

   The fix has been deployed to production, and visibility within the system has been restored. We’re monitoring closely to ensure everything is working as expected. Your Sympa Team

4. resolved Nov 06, 2025, 06:11 PM UTC

   The issue has been fully resolved. System visibility and access are now working as expected. Thank you for your patience while we implemented the fix. Your Sympa Team

Read the full incident report →

Started Oct 29, 2025, 05:35 PM UTC· resolved Oct 30, 2025, 05:53 AM UTC

1. identified Oct 29, 2025, 05:35 PM UTC

   We have identified an issue that has impacted Sympa. This is unfortunately part of an Global Azure Incident: https://azure.status.microsoft/en-gb/status -------- Starting at approximately 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state. We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly. We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update. This message was last updated at 17:18 UTC on 29 October 2025 -------- At the moment we can only monitor the situation from Sympa's point of view. We’ll share updates as soon as more information becomes available.

2. identified Oct 29, 2025, 06:16 PM UTC

   Latest status update from Azure: https://azure.status.microsoft/en-us/status ----- Starting at approximately 16:00 UTC, customers and Microsoft services that leverage Azure Front Door (AFD) may have experienced issues resulting in latencies, timeouts and errors. We have confirmed that an inadvertent configuration change as the trigger event for this issue. Current status: We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes. Customer configuration changes will remain blocked during this time as we work towards mitigation. We will communicate to customers when this block is reverted. Customers may have experienced problems accessing the Azure management portal. We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly, while all portal extensions are working correctly there may be a small number of endpoints that might have a problem loading (i.e. Marketplace). We do not have an ETA for full mitigation, we will update this communication within 30 minutes, once the deployment is completed. Customers can consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins This message was last updated at 18:11 UTC on 29 October 2025 -----

3. identified Oct 29, 2025, 07:13 PM UTC

   Latest status update from Azure: https://azure.status.microsoft/en-us/status ----- Current status: We have pushed our ‘last known good’ configuration, and customers may begin to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement. Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted. Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems. We are continuing to monitor progress closely and will provide an ETA for full mitigation within the next 20 minutes as we assess recovery across the AFD service. Although we are seeing signs of recovery, customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure. https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview This message was last updated at 19:01 UTC on 29 October 2025 -----

4. identified Oct 29, 2025, 07:56 PM UTC

   Status from Microsoft's side is not yet closed but they expect situation to continue to improve. We continue to monitor our system as we wait for mitigation to take effect. Latest status update from Azure: https://azure.status.microsoft/en-us/status ----- We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement. At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted. -----

5. resolved Oct 30, 2025, 05:53 AM UTC

   This incident has been resolved. A short recap from Microsoft's point of view: ----- What went wrong and why? An inadvertent tenant configuration change within Azure Front Door (AFD) triggered a widespread service disruption affecting both Microsoft services and customer applications dependent on AFD for global content delivery. The change introduced an invalid or inconsistent configuration state that caused a significant number of AFD nodes to fail to load properly, leading to increased latencies, timeouts, and connection errors for downstream services. As unhealthy nodes dropped out of the global pool, traffic distribution across healthy nodes became imbalanced, amplifying the impact and causing intermittent availability even for regions that were partially healthy. We immediately blocked all further configuration changes to prevent additional propagation of the faulty state and began deploying a ‘last known good’ configuration across the global fleet. Recovery required reloading configurations across a large number of nodes and rebalancing traffic gradually to avoid overload conditions as nodes returned to service. This deliberate, phased recovery was necessary to stabilize the system while restoring scale and ensuring no recurrence of the issue. The trigger was traced to a faulty tenant configuration deployment process. Our protection mechanisms, to validate and block any erroneous deployments, failed due to a software defect which allowed the deployment to bypass safety validations. Safeguards have since been reviewed and additional validation and rollback controls have been immediately implemented to prevent similar issues in the future. Our team will be completing an internal retrospective to understand the incident in more detail and will share findings within 14 days. Once we complete our internal retrospective, generally within 14 days, we will publish a final Post Incident Review (PIR) to all impacted customers. ----- Best regards Sympa Team

Read the full incident report →

Started Oct 10, 2025, 05:57 AM UTC· resolved Oct 10, 2025, 06:59 AM UTC

1. monitoring Oct 10, 2025, 05:57 AM UTC

   Dear Customer, We are currently experiencing general degraded performance across the Sympa platform. Our team has identified increased slowness affecting multiple areas of the system and is actively monitoring the situation. We are working to ensure the performance stabilizes as soon as possible and will provide updates as we make progress. Best regards, Sympa Team

2. resolved Oct 10, 2025, 06:59 AM UTC

   Dear Customer, The issue causing general slowness across the Sympa platform has now been fully resolved. Our team identified the cause and took corrective actions that have restored normal performance levels. We will still continue to monitor the system closely, but all services are now operating as expected. We apologize for any inconvenience this may have caused and thank you for your patience and understanding. Best regards, Sympa Team

Read the full incident report →

Started Aug 26, 2025, 05:30 AM UTC· resolved Aug 26, 2025, 08:04 AM UTC

1. identified Aug 26, 2025, 05:30 AM UTC

   We have identified an issue at 26.Aug.2025 at 8:15 EEST that may have impacted Sympa. We are working to resolve the problem and will provide an update when it is confirmed to be fully resolved.

2. monitoring Aug 26, 2025, 05:54 AM UTC

   At 26 Aug 2025, 8:52 EEST a fix has been implemented and the system is operational. We're currently monitoring the results.

3. resolved Aug 26, 2025, 08:04 AM UTC

   The incident has been resolved. There was network disruption between 06:00 - 08:37 that affected small portion of API calls, appearing as randomly failing requests and errors shown to the user.

Read the full incident report →

Started Aug 15, 2025, 05:52 AM UTC· resolved Aug 15, 2025, 08:10 AM UTC

1. investigating Aug 15, 2025, 05:52 AM UTC

   Our team is currently investigating why attachments can not be downloaded neither previewed in Sympa. At the moment attachments under any person cannot be previewed neither downloaded.

2. monitoring Aug 15, 2025, 06:52 AM UTC

   We want to inform you about a recent technical issue in our system that occurred between 14/08/2025 at 17:00 EEST and 15/08/2025 at 09:05 EEST. During this period, some attachments uploaded via the Sympa UI (e.g., over the user profile) may not have been saved correctly. The issue has been fully resolved, and the system is now operating normally. Only a limited number of customers were affected. We will contact all affected customers directly with further details and instructions. If you do not hear from us, no action is required on your part. We sincerely apologize for any inconvenience this incident may have caused and thank you for your understanding. Your Sympa Team

3. resolved Aug 15, 2025, 08:10 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 08, 2025, 08:28 AM UTC· resolved Aug 08, 2025, 08:28 AM UTC

1. resolved Aug 08, 2025, 08:28 AM UTC

   We experienced an issue affecting file-based integrations, resulting in degraded functionality. Start time: 21:18 (07/08/2025) Resolved: 11:20 (08/08/2025) The problem has now been resolved. We sincerely apologize for any inconvenience this may have caused and appreciate your patience and understanding. If you have any questions or need further assistance, please don't hesitate to contact us.

Read the full incident report →

Started Jul 08, 2025, 07:07 AM UTC· resolved Jul 08, 2025, 07:25 AM UTC

1. investigating Jul 08, 2025, 07:07 AM UTC

   We are currently investigating an increased number of network errors affecting connections from users’ browsers to Sympa. Some users may experience difficulty accessing the service or notice intermittent connectivity issues. Our team is actively working to identify the root cause and resolve the issue as quickly as possible. We apologize for any inconvenience and appreciate your patience. Further updates will be posted here as soon as more information is available.

2. resolved Jul 08, 2025, 07:25 AM UTC

   The issue causing network errors and intermittent connectivity for some users has been identified and resolved. The root cause was a broken certificate, which has now been replaced. All services are operating normally again. We sincerely apologize for the disruption and appreciate your patience while we worked to restore full functionality. If you continue to experience any issues, please reach out to our support team.

Read the full incident report →

Started Jun 23, 2025, 08:37 AM UTC· resolved Jun 23, 2025, 12:56 PM UTC

1. investigating Jun 23, 2025, 08:37 AM UTC

   There is currently heavy load in our systems and our team is investigating what's causing.

2. monitoring Jun 23, 2025, 08:41 AM UTC

   Our team has found the reason behind the slowness and we're now monitoring the situation.

3. monitoring Jun 23, 2025, 09:27 AM UTC

   Sympa Performance Notice We have identified performance issues in the Sympa system, which are linked to specific types of data imports affecting database performance. Our team is actively investigating the root cause to ensure a permanent resolution. As a precautionary measure, we have temporarily paused the affected data import processes to prevent further impact. We will provide updates as soon as more information becomes available or once the issue has been fully resolved. We apologize for any inconvenience this may cause and appreciate your patience and understanding. Please note, this is not affecting integrations.

4. resolved Jun 23, 2025, 12:56 PM UTC

   System is working again normally.

Read the full incident report →

Started Jun 04, 2025, 12:09 PM UTC· resolved Jun 04, 2025, 05:42 PM UTC

1. identified Jun 04, 2025, 12:09 PM UTC

   We have identified an issue with our Support Knowledgebase provider Hubspot that causes an error for some users when trying to login to the Knowledgebase. We will update you once the issue is solved.

2. resolved Jun 04, 2025, 05:42 PM UTC

   The sign in page for knowledge base login was fixed and login into knowledge works again.

Read the full incident report →

Started May 21, 2025, 03:30 AM UTC· resolved May 21, 2025, 05:10 AM UTC

1. investigating May 21, 2025, 04:51 AM UTC

   At 3:30 UTC We have identified an issues with the IAM login, where some customers using that sign in method are unable to login. This can effect both admins and endusers at the moment. We are currently investigating the issue with priority. We we'll keep you updated on the progress.

2. resolved May 21, 2025, 05:10 AM UTC

   At 5:05 UTC the problem was resolved and all Login is now operational. If the users have tried logging in during the disruption the browser cache might require cleaning.

3. postmortem May 21, 2025, 10:02 AM UTC

   On the evening of **May 20th at 21:30 \(EEST\)**, a change to our **Content Delivery Network \(CDN\)** caused a disruption affecting some users of our new login system. The issue was resolved by **08:20 the following morning**, May 21st. The disruption was due to a configuration error introduced during the CDN change. This was identified and fixed by **08:20 \(EEST\)** the following morning. This CDN improvement is part of our efforts to strengthen security and make it easier for us to update our critical infrastructure moving forward. We’re very sorry for the inconvenience this may have caused and appreciate your patience as we continue working to provide a reliable and secure service. If you have any questions or need assistance, our support team is here to help.

Read the full incident report →

Started May 08, 2025, 06:34 PM UTC· resolved May 10, 2025, 06:52 AM UTC

1. investigating May 08, 2025, 06:34 PM UTC

   There is currently an issue with the latest update where users aren't able to access Sympa. We are investigating the issue.

2. monitoring May 08, 2025, 06:43 PM UTC

   An issue was found and fixed. We are monitoring the situation.

3. resolved May 10, 2025, 06:52 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 08, 2025, 02:50 PM UTC· resolved Apr 08, 2025, 04:55 PM UTC

1. investigating Apr 08, 2025, 02:50 PM UTC

   We are currently investigating an issue where some users are experiencing problems when attempting to create new datacards. Our team is actively working to identify the root cause and will provide updates as soon as more information is available. Thank you for your patience.

2. resolved Apr 08, 2025, 04:55 PM UTC

   Our engineering team found out the issue and it's now solved, production working again normally.

3. postmortem Apr 28, 2025, 06:10 AM UTC

   Between 4.00pm EEST and 7.30pm EEST, creation of new employees and data cards failed. The root cause was identified to be in the backend service of the system, which had a limitation, and prevented the operation to complete. The situation was fixed on-the-fly by removing the limitation, which fixed the situation. Removing the limitation, allows system to work normally, but a proper software change will be done to ensure we are not going to hit the same limitation in the future.

Read the full incident report →

Started Mar 24, 2025, 10:44 AM UTC· resolved Mar 25, 2025, 01:38 PM UTC

1. identified Mar 24, 2025, 10:44 AM UTC

   We are currently experiencing issues with the e-signing functionality, which is being impacted by a disruption with our third-party provider. Our team is actively monitoring the situation and working with the provider to restore full service as quickly as possible. We will continue to provide updates as we receive more information. Thank you for your patience.

2. identified Mar 24, 2025, 03:14 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Mar 24, 2025, 03:15 PM UTC

   The issue is identified to affect only VismaSign and the problem is on the Visma side. We have notified the provider and we are monitoring for updates on their side.

4. monitoring Mar 25, 2025, 11:10 AM UTC

   We've solved the issue together with the 3rd party and necessary changes have been deployed to production systems yesterday afternoon. We're still monitoring the situation.

5. resolved Mar 25, 2025, 01:38 PM UTC

   The issue has been resolved and VismaSign should now be fully functional.

6. postmortem Mar 25, 2025, 03:41 PM UTC

   Between 21 February at 12:00 EET and 24 March at 13:40 EET, users using VismaSign experienced issues with the e-signing functionality due to a disruption on the VismaSign side. According to VismaSign and their partner Twoday, the issue was caused by an expired certificate, which has since been renewed. Logs indicate that traffic returned to normal after the fix, and new documents have been created successfully since. The service is now fully operational. Please let us know if you continue to experience any issues.

Read the full incident report →

Started Mar 21, 2025, 10:43 AM UTC· resolved Mar 21, 2025, 11:00 AM UTC

1. identified Mar 21, 2025, 10:43 AM UTC

   We are currently investigating an issue where some users may experience problems saving data in the system. Our team is actively working to identify the cause and implement a fix. We will provide an update as soon as we have more information. Thank you for your patience.

2. resolved Mar 21, 2025, 11:00 AM UTC

   We've identified the cause of the issue affecting data saving in the system. It was due to an operation which impacted the ability to save data. The problem has now been solved, and the system is functioning normally again. We appreciate your patience during this time.

3. postmortem Mar 21, 2025, 02:13 PM UTC

   **POST MORTEM: DISRUPTION 21st of March, Issues with Saving data** On March 21st, 2025, between 12:10 – 12:49 EET, an increase in system errors caused disruptions for some customers. During this time, users experienced issues saving data and delays in contracts being sent. The issue was identified at 12:30 EET, triggered by an alert following increased error rates. It was traced back to an operation performed in one of our databases, which unintentionally blocked changes to it. The blockage was resolved at 12:49 EET. **Timeline** * **12:10 EET** – The issue began, leading to increased errors and disruptions in saving data and sending contracts. * **12:30 EET** – The issue was detected by our monitoring. * **12:49 EET** – The underlying cause was addressed and error rates dropped **Preventative Measures** To reduce the risk of similar issues in the future, we are implementing the following improvements: * Introducing more specific alerts to quickly detect database operations that may cause system disruptions, allowing faster investigation and resolution. * Further limiting certain internal operations during working hours to reduce the risk of impact on active users. We sincerely apologize for the inconvenience and appreciate your understanding as we work to continuously improve our systems.

Read the full incident report →

Started Feb 12, 2025, 06:50 AM UTC· resolved Feb 12, 2025, 06:50 AM UTC

1. resolved Feb 12, 2025, 08:58 AM UTC

   We experienced a temporary issue today between 8:50 and 9:15 AM, which caused signed-in users to be unexpectedly logged out. As a result, affected users encountered access error pages when navigating the system and were unable to log back in during this time. The issue has since been resolved, and normal login functionality has been restored.

Read the full incident report →

Started Dec 13, 2024, 10:00 AM UTC· resolved Dec 13, 2024, 07:14 PM UTC

1. investigating Dec 13, 2024, 10:00 AM UTC

   We are currently investigating an issue impacting the Approvals functionality. Our team is actively working to identify the cause and resolve the issue as quickly as possible. We will share updates as we have more information. Thank you for your patience and understanding.

2. identified Dec 13, 2024, 10:56 AM UTC

   We have identified root cause for the issue and are working on a hotfix. We will share updates as we have more information. Thank you for your patience and understanding.

3. monitoring Dec 13, 2024, 01:42 PM UTC

   A fix has been implemented and we are monitoring the situation.

4. resolved Dec 13, 2024, 07:14 PM UTC

   The issue impacting the Approvals functionality has been fully resolved. Our team has implemented a fix, and monitoring confirms that the functionality is working as expected. We appreciate your patience and understanding during this time. If you continue to experience any issues, please reach out to our support team.

5. postmortem Dec 16, 2024, 07:17 PM UTC

   **Post-Mortem Report: Production Deployment Issue on 12.12** **Incident Overview:** On 12.12 at 21:00, a new software package was deployed to production. During this deployment, two issues were introduced, both related to approval message functionality. These issues impacted the timely construction and delivery of approval-related emails. **Details of the Issues:** **1. Missing Database Access Rights** * **Impact**: This issue caused the failure of all approval message constructions. * **Timeline**: The problem persisted between 12.12 at 21:50 and 13.12 at 9:30. * **Mitigation**: The issue was resolved by granting the necessary database access rights at 13.12 at 9:30. * **Effect**: No approval emails were sent during the affected period. **2. Undefined Sympa Username for Related Persons** * **Impact**: Approval messages failed to be sent in cases where one or more related individuals did not have a Sympa username defined. This affected all recipients involved in the corresponding approval. * **Timeline**: This issue occurred from 13.12 at 9:30 to 13.12 at 15:30. * **Mitigation**: The underlying issue was identified and corrected by 13.12 at 15:30. * **Effect**: Emails associated with these specific approvals were not sent. **Scope of Impact** The issue impacted approval messages across multiple customers, resulting in a significant number of unprocessed emails. In total, the deployment failure affected approximately 450 approval messages across 60 customers, where most of customer only few messages concerned. These numbers illustrate the widespread nature of the problem during the incident. **Technical Constraints** Due to the nature of the issue, which disrupted the construction of approval messages, it is unfortunately not possible to reconstruct and resend these emails automatically. This limitation stems from the absence of retained data necessary for reconstructing the messages in their original context. **Next Step** Sympa can provide a detailed list of the affected approvals upon request. This list can assist in manually addressing any critical messages that were not delivered. **Preventative Measures** To prevent similar issues in the future, the following actions will be taken: * Implementing more rigorous pre-deployment testing, specifically targeting approval-related functionality. * Establishing automated monitoring for database access configurations to detect and rectify issues prior to deployment. * Enhancing error logging and data retention mechanisms to allow reconstruction of critical messages if failures occur. We apologize for the inconvenience caused and remain committed to improving the reliability of our systems.

Read the full incident report →

Started Nov 19, 2024, 12:09 PM UTC· resolved Nov 21, 2024, 08:56 AM UTC

1. monitoring Nov 19, 2024, 12:09 PM UTC

   Microsoft Azure is currently performing maintenance on App Services. As a result, you may notice temporary connection drops or some slowness when accessing services. We are monitoring the situation and will provide updates as needed. Thank you for your patience.

2. resolved Nov 21, 2024, 08:56 AM UTC

   The scheduled maintenance by Microsoft Azure on App Services has been successfully completed. All services are now operating normally, and no further connection issues or slowness are expected. We appreciate your patience during this time. Thank you for your understanding.

Read the full incident report →