Sympa HR incident

Integration processing disruption

Resolved after 22 hours, affecting Integrations, on June 16, 2026.

postmortem Jul 01, 2026, 02:12 PM UTC

## Post mortem for customers ### What happened Some integrations were delayed or failed during the nightly processing window because Azure Blob Storage operations were slower than usual. The integrations performed more Azure Blob listing/checking operations than expected. The number of `ListBlobs` calls was around double compared to normal days. This affected to one integration application. ### Why it happened Most probably, Azure Blob Soft Delete or increased blob volume caused slower listing operations during the night. This affected integrations that rely on file lookup logic to Azure Blob. ### What we did We optimized Azure Blob existence checks and flows in the integration application that got affected. As a result, the number of `ListBlobs` calls decreased by around 20%, and the next nightly integration runs completed normally, with better performance. The same fix was also applied to other integration applications.

Started
Jun 16, 2026, 08:17 AM UTC
Resolved
Jun 17, 2026, 06:17 AM UTC
Duration
22h
Detected by Pingoru
Jun 16, 2026, 08:17 AM UTC

Affected components

Integrations

Update timeline

  1. investigating Jun 16, 2026, 08:17 AM UTC

    Dear Customer, We are currently investigating an issue affecting integrations. Customers may experience failed or delayed integration processing. Our team is actively investigating the root cause and working to restore normal integration processing. We will provide the next update by 13:00 EEST, or sooner if there is significant progress. We apologize for the disruption and appreciate your patience. Your Sympa Team

  2. identified Jun 16, 2026, 10:01 AM UTC

    Dear Customer, We have identified the cause of the issue affecting integrations. Customers may still experience delayed integration processing. Our team is continuing mitigation work and manually reprocessing affected files where possible while we work to restore normal processing. We will provide the next update by 15:00 EEST, or sooner if there is significant progress. Your Sympa Team

  3. identified Jun 16, 2026, 12:14 PM UTC

    Dear Customer, We are continuing work to restore normal integration processing. Customers may still experience delayed integration processing. Our team is preparing a fix and testing it before applying it. We are also continuing mitigation work and manually reprocessing affected files where possible. We will provide the next update by 16:30 EEST, or sooner if there is significant progress. Your Sympa Team

  4. monitoring Jun 16, 2026, 01:36 PM UTC

    Dear Customer, We have applied fixes for the issue affecting integrations and are monitoring their effect. Integration processing has improved, but some data from last night still requires manual handling. Our team is continuing this work and will monitor the results of the fixes throughout the night. We will keep this incident open and provide the next update by 19:00 EEST, or sooner if there is significant progress. Your Sympa Team

  5. monitoring Jun 16, 2026, 04:00 PM UTC

    Dear Customer, We have applied mitigation actions for the issue affecting integrations and are continuing to monitor integration processing. Our team is monitoring the evening and overnight processing runs to confirm stability. We are also continuing manual handling of any remaining affected data where needed. We will keep this incident open while monitoring continues and will provide the next update by 09:00 EEST tomorrow, or sooner if there is significant progress. Your Sympa Team

  6. resolved Jun 17, 2026, 06:17 AM UTC

    The mitigation actions taken yesterday has fixed the issue. The performance has also improved significantly. We did not detected such issues in integrations during yesterday evening, during the night nor during the morning. We confirm everything is operational and close this ticket. Post-mortem will be released later on.

  7. postmortem Jul 01, 2026, 02:12 PM UTC

    ## Post mortem for customers ### What happened Some integrations were delayed or failed during the nightly processing window because Azure Blob Storage operations were slower than usual. The integrations performed more Azure Blob listing/checking operations than expected. The number of `ListBlobs` calls was around double compared to normal days. This affected to one integration application. ### Why it happened Most probably, Azure Blob Soft Delete or increased blob volume caused slower listing operations during the night. This affected integrations that rely on file lookup logic to Azure Blob. ### What we did We optimized Azure Blob existence checks and flows in the integration application that got affected. As a result, the number of `ListBlobs` calls decreased by around 20%, and the next nightly integration runs completed normally, with better performance. The same fix was also applied to other integration applications.

Your product runs on more than Sympa HR

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →