AskCody incident

[Europe] Degraded performance - Management portal, Add-ins, check-in kiosks

Major Resolved View vendor source →

AskCody experienced a major incident on August 18, 2025 affecting Outlook Add-in and Outlook Add-in and 1 more component, lasting 20h 18m. The incident has been resolved; the full update timeline is below.

Started
Aug 18, 2025, 12:55 PM UTC
Resolved
Aug 19, 2025, 09:13 AM UTC
Duration
20h 18m
Detected by Pingoru
Aug 18, 2025, 12:55 PM UTC

Affected components

Outlook Add-inOutlook Add-inOutlook Add-inVisitor Management PortalMeeting Services PortalRoom Management PortalMeeting DashboardsCheck-in kioskRoom DisplaysMeeting Services Finance API

Update timeline

  1. investigating Aug 13, 2025, 01:06 PM UTC

    Affected Users: [Add-in Users, Management Portal Users,.] Region: [Europe] We are currently experiencing a partial outage in the management portal and Outlook Add-ins and are investigating the issue. Users may experience this issue as timeouts and slow response times. As a temporary workaround, users can retry their requests after a few seconds. The next update will be provided as soon as possible.

  2. investigating Aug 13, 2025, 01:06 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Aug 13, 2025, 02:58 PM UTC

    Affected Users: [Add-in Users, Management Portal Users] Region: [Europe] We have identified and fixed the issue and we are actively monitoring the platform.

  4. identified Aug 14, 2025, 07:22 AM UTC

    Affected Users: [Add-in Users, Management Portal Users,.] Region: [Europe] We can see that the issue persists and we are working on a solution. You will as a user continue to experience long loading times and timeouts. It will, in most cases help if you refresh the page/add-in. The next update will be provided as soon as we know more.

  5. identified Aug 14, 2025, 11:24 AM UTC

    Affected Users: [Add-in Users, Management Portal Users,.] Region: [Europe] We can see that the issue persists and we are working on a solution. You will as a user continue to experience long loading times and timeouts. It will, in most cases help if you refresh the page/add-in. We have a support case going with Microsoft to try and resolve this and will continue to work on the issue until it has been resolved. The next update will be provided as soon as we know more.

  6. identified Aug 15, 2025, 06:39 AM UTC

    Affected Users: [Add-in Users, Management Portal Users,.] Region: [Europe] We are still trying to solve this. You will as a user continue to experience long loading times and timeouts. It will, in most cases help if you refresh the page/add-in. We are in dialogue with Microsoft and will continue to work on the issue until it has been resolved.

  7. identified Aug 15, 2025, 08:32 AM UTC

    Affected Users: [Add-in Users, Management Portal Users,.] Region: [Europe] We are aware that access to platform disappeared entirely and expect it to be back shortly.

  8. monitoring Aug 15, 2025, 09:13 AM UTC

    Affected Users: [Add-in Users, Management Portal Users,.] Region: [Europe] The platform currently appears to be fully operational again and we are monitoring if this continues to be the case. You should not experience timeouts and loading errors currently.

  9. investigating Aug 15, 2025, 09:48 AM UTC

    We can see that the issue persists and we are working on a solution. You will as a user continue to experience long loading times and timeouts. It will, in most cases help if you refresh the page/add-in. The next update will be provided as soon as we know more.

  10. investigating Aug 15, 2025, 12:51 PM UTC

    The issue persists, and we are continuing to investigate possible causes. Users may still experience long loading times and timeouts. In many cases, refreshing the page or add-in can help temporarily. We are working to restore normal performance as soon as possible and will provide the next update once we have more information.

  11. investigating Aug 18, 2025, 07:06 AM UTC

    We are unfortunately still not able to resolve this issue and while we work towards a solution, we are also trying to help mitigate the situation while we solve it by making the platform available to you without errors. We hope to install a small fix that should help provide access but with slow loading times (no timeouts) soon. Users may still experience long loading times. In many cases, refreshing the page or add-in can help temporarily. We are working to restore normal performance as soon as possible and will provide the next update once we have more information.

  12. investigating Aug 18, 2025, 12:55 PM UTC

    We are still working to fully resolve this issue. In the meantime, we are taking steps to mitigate the impact and ensure you can continue using the platform without errors. A small fix will soon be deployed to improve access. While this may result in slower loading times, it should prevent timeouts. Please note that users may still experience delays. Refreshing the page or add-in often helps temporarily. We are committed to restoring normal performance as quickly as possible and will share another update as soon as we have more information.

  13. investigating Aug 18, 2025, 01:29 PM UTC

    We are still working to fully resolve this issue. In the meantime, we are taking steps to mitigate the impact and ensure you can continue using the platform without errors. A small fix will soon be deployed to improve access. While this may result in slower loading times, it should prevent timeouts. Please note that users may still experience delays. Refreshing the page or add-in often helps temporarily. We are committed to restoring normal performance as quickly as possible and will share another update as soon as we have more information.

  14. monitoring Aug 18, 2025, 01:58 PM UTC

    We have identified and isolated the issue affecting the platform, and it should now be operating normally. While the platform is accessible normally again, some users may continue to experience slower loading times. In these cases, refreshing the page or add-in will help. We are continuing to monitor performance closely and will provide further updates if needed.

  15. resolved Aug 19, 2025, 09:13 AM UTC

    Affected Users: [Add-in Users, Management Portal Users] Region: Europe After monitoring the implemented solution, the issue related to the slowness/timeouts and the degraded performance has been resolved, and the incident will now be closed. The identified faulty endpoint has been isolated and the platform is operational again.

  16. postmortem Aug 19, 2025, 10:40 AM UTC

    # \[EU\] Latency and timeouts from secondary services - Post Mortem ‌ This Postmortem and technical timeline follows the announcement of degraded performance on the AskCody platform, beginning on Wednesday, August 13th. ## Summary On August 13th at 13:12 CEST, parts of the AskCody platform became unresponsive with very long response times and timeout errors. The errors were periodic and were not caused by any changes made by AskCody. This report describes the timeline and remediation steps taken to resolve the issue. ## Incident overview On August 13th, 2025, at 13:15 CEST, our team was alerted to a severe drop in availability on the [app.onaskcody.com](http://app.onaskcody.com/) domain, serving both our management portal, add-ins, and the check-in kiosks to our EU customers. The alert was in response to a steep increase in response times that had manifested a few minutes earlier. ## Investigation and actions taken ### Initial response August 13, at 13:15 CEST: Detected drop in availability and influx in response times. August 13, at 13:15 CEST: Extended team investigation into the cause and effects of the incident. No conclusive evidence. ### Investigation, temporary solutions, and monitoring May 14 - 15: Investigations extended to include Microsoft Azure services as PaaS resources seemed likely to be involved. Working with Microsoft to profile latency through their network and services. ### Critical discovery August 18, 13:00 CEST: Tracking led to the identification of periodically failing calls to a third-party service that had made changes to their ingestion on the 13th. ### Resolution August 18,. 14:00 CEST: The issue was mitigated, and the incident status was set to “monitoring”. ## Root cause analysis The root cause of the incident was a misconfigured request to a third-party service. We had received timely advice to alter the configuration, but the fact that the endpoint still periodically accepted requests made the troubleshooting more complex. ## Preventative and supplementary measures To prevent similar incidents in the future, we have: * Extended alerting systems for availability tests. ## Conclusion Though this incident was not classified as **critical**, we still consider the impact to be serious. We apologize for any inconvenience caused by the service degradation and assure you that our platform is fully operational again. We remain committed to maintaining the highest security standards, following Microsoft best practices, and will continue to take proactive measures to safeguard our services. If you have any questions or need further information, please do not hesitate to contact our support team. Thank you for your understanding and continued trust in AskCody. Sincerely, **AskCody**