Orchestry Outage History

Started Oct 15, 2026, 02:51 PM UTC· resolved Oct 15, 2026, 08:46 PM UTC

1. investigating Oct 15, 2026, 02:51 PM UTC

   We’re aware of an issue causing File Archival jobs to fail for some customers in our US region. Affected jobs fail during execution, and archival actions scheduled during this period may not have completed. We’ve traced the cause to a recent change in Microsoft SharePoint Online that affects how certain search requests are processed. Microsoft has acknowledged reports of the problem, which appears to be impacting a subset of SharePoint Online tenants. Our team is investigating if the issue can be worked around on our side so that File Archival no longer depends on the affected behavior, and we’re also monitoring Microsoft’s progress on a resolution. No files have been lost or modified as a result of this issue; jobs that failed simply did not run. We’ll post another update as soon as we have more information or a fix is deployed.

2. identified Oct 15, 2026, 04:21 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 15, 2026, 07:44 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 15, 2026, 08:46 PM UTC

   This incident has been resolved.

5. postmortem Oct 16, 2026, 06:53 PM UTC

   **What happened:** From October 14 to 15, File Archival jobs failed for some customers in our US region. Scheduled archival actions during this time did not run. No files were lost or modified. **Detection:** Our monitoring caught the failures early, which allowed us to begin investigating right away. **Why:** A change Microsoft made to SharePoint Online caused some of the search requests File Archival relies on to be rejected. It affected only some SharePoint tenants. **How we fixed it:** We updated File Archival to work with the change and deployed the fix on October 15. Jobs are completing normally again. We apologize for the disruption. If you have questions, please contact our support team.

Read the full incident report →

Started Aug 17, 2026, 02:26 PM UTC· resolved Aug 17, 2026, 10:06 PM UTC

1. investigating Aug 17, 2026, 02:26 PM UTC

   We are experiencing delays in background processing across all regions. Some customers may see workspace provisioning and reporting data updates take longer than usual or appear stuck in progress. The Orchestry app itself remains available. We have identified the cause and are working on remediation. No data has been lost, delayed work will complete once processing recovers.

2. identified Aug 17, 2026, 05:34 PM UTC

   We have confirmed the cause of the background processing delays and are working towards a fix. Affected job servers are being restarted along with a temporary mitigation to restore throughput while we finalize the fix.

3. monitoring Aug 17, 2026, 09:28 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 17, 2026, 10:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 21, 2026, 02:49 PM UTC· resolved Jul 22, 2026, 02:40 PM UTC

1. identified Jul 21, 2026, 02:49 PM UTC

   We're aware that some people are experiencing failures when deploying SPFx packages to the SharePoint tenant app catalog using app-only (certificate/application) authentication. Affected requests return the following error, even when the app registration has Sites.FullControl.All granted with admin consent: "Must have Manage Web Site permissions in order to add apps to site collection app catalog" This issue is currently preventing automatic updates to the web part package for Orchestry users. We have reported this to Microsoft and are tracking it closely. This appears to be a Microsoft-side issue, not something within Orchestry's control. It has been reported to Microsoft and is being tracked publicly: PnP PowerShell GitHub issue #5406 [https://github.com/pnp/powershell/issues/5406] SharePoint dev docs GitHub issue #10955 — confirmed as a bug by Microsoft [https://github.com/SharePoint/sp-dev-docs/issues/10955] We are monitoring both threads closely and will post updates here as we learn more. If you're impacted, please reach out to our Customer Success team at support.orchestry.com for a manual workaround.

2. resolved Jul 22, 2026, 02:40 PM UTC

   Microsoft has confirmed that a mitigation has been put in place to resolve the issue affecting SPFx package uploads to the SharePoint tenant app catalog via app-only (certificate/application) authentication. Multiple customers have confirmed the fix in Microsoft's public tracking thread, and our own internal testing confirms the issue no longer reproduces. If you were previously impacted and continue to see this error, please reach out to our Customer Success team at support.orchestry.com. Thanks for your patience while we tracked this down alongside Microsoft.

Read the full incident report →

Started Jun 16, 2026, 03:19 PM UTC· resolved Jun 16, 2026, 09:27 PM UTC

1. identified Jun 16, 2026, 03:19 PM UTC

   **Investigating – Workspace Provisioning and Template Application Failures** We are currently investigating an issue affecting workspace provisioning, template application, and certain installation/update operations for some customers in Orchestry. Our investigation has identified that the issue is being caused by a recent regression in a Microsoft SharePoint API that Orchestry relies on during site provisioning and template deployment. As a result, customers may experience failures when: * Creating new workspaces * Creating Microsoft Teams workspaces * Applying templates to new or existing sites * Running Orchestry installations or updates * Performing archive operations that create destination sites Our team has identified a mitigation and is actively working to deploy a hotfix that bypasses the affected Microsoft API call. We are continuing to monitor the situation and will provide additional updates as more information becomes available. We apologize for the inconvenience and appreciate your patience while we work to resolve this issue.

2. identified Jun 16, 2026, 06:45 PM UTC

   Hotfix Deployment in Progress We have identified the root cause of the issue impacting workspace provisioning, template application, and related operations. Our investigation indicates that a recent regression in a Microsoft SharePoint API is causing failures when Orchestry interacts with the API through the PnP framework. This issue affects functionality that configures workspace header and footer settings during provisioning and template deployment. To restore service as quickly as possible, we are deploying a hotfix that temporarily bypasses workspace header and footer configuration. This mitigation addresses the most critical customer impact by allowing workspace provisioning and related operations to complete successfully. While this hotfix restores affected functionality, our team continues to investigate a long-term solution for the underlying Microsoft API regression and will provide further updates as they become available. We appreciate your patience and understanding as we work through this issue.

3. monitoring Jun 16, 2026, 07:34 PM UTC

   Monitoring – Hotfix Successfully Deployed We have successfully deployed a hotfix and have verified that workspace provisioning, template application, and related operations are functioning as expected. The hotfix mitigates the immediate impact of the issue by temporarily bypassing workspace header and footer configuration, which was triggering failures due to an underlying regression affecting Microsoft SharePoint APIs when accessed through the PnP framework. We are actively monitoring the platform to confirm continued stability and successful operation of affected workflows. Our investigation into the underlying issue remains ongoing. We are working toward a long-term solution that restores full workspace header and footer configuration functionality without requiring this temporary bypass. We will continue to provide updates as more information becomes available.

4. resolved Jun 16, 2026, 09:27 PM UTC

   Resolved – Workspace Provisioning Issue The issue affecting workspace provisioning, template application, and related operations has been resolved. We have deployed a mitigation that bypasses workspace header and footer configuration during provisioning, allowing affected operations to complete successfully while relying on the default SharePoint header and footer configuration. We have verified that provisioning and related workflows are functioning normally following this change. Our investigation determined that the underlying issue is related to a regression affecting Microsoft SharePoint APIs when accessed through the PnP framework. While the immediate customer impact has been resolved, we continue to work on a permanent fix that will restore full workspace header and footer configuration functionality. We expect to deliver this enhancement as part of our normal release cadence, with the current target being our next scheduled release on June 18, 2026, pending successful validation and testing. Thank you for your patience and understanding throughout this incident.

5. postmortem Jun 23, 2026, 09:35 PM UTC

   **Resolved – Workspace Provisioning Issue** The issue affecting workspace provisioning, template application, and related operations has been fully resolved. Our investigation determined that the underlying issue was caused by a regression affecting Microsoft SharePoint APIs when accessed through the PnP framework. This regression caused header and footer configuration operations to fail during provisioning, resulting in unsuccessful workspace creation and template deployment. As an immediate mitigation, we previously deployed a hotfix that bypassed workspace header and footer configuration, allowing provisioning and related operations to complete successfully while relying on default SharePoint settings. We have now released a permanent fix that restores workspace header and footer configuration while avoiding the affected Microsoft API behavior. The solution leverages an alternative implementation path that preserves provisioning reliability while ensuring workspace branding and navigation settings are applied correctly. We have verified the fix across affected provisioning scenarios, and all impacted functionality is operating normally.

Read the full incident report →

Started Jan 08, 2026, 03:54 PM UTC· resolved Jan 20, 2026, 03:44 PM UTC

1. monitoring Jan 08, 2026, 03:54 PM UTC

   We are currently monitoring an issue affecting workspace archivals that include certain group configuration options. Archival policies may fail when either of the following options is enabled: * Remove from Global Address List (GAL) * Remove from Exchange Client This issue is caused by a recent change in Microsoft’s Microsoft Graph service and is external to Orchestry. As a result, affected archival jobs may appear to succeed but do not complete all archival actions. Impact * Some archival jobs may not complete as expected. * No data loss has occurred. Workaround As a temporary measure, we recommend disabling the above group configuration options in archival policies until Microsoft resolves the underlying issue. Our engineering team has identified the root cause and is actively monitoring Microsoft’s progress on a fix. We will provide updates as more information becomes available.

2. monitoring Jan 09, 2026, 03:45 PM UTC

   This problem persists. We have opened a support ticket with Microsoft and they are aware of the issue but appear to be still investigating. We will continue to monitor and add updates here as we find out more details.

3. monitoring Jan 13, 2026, 06:14 PM UTC

   This issue remains open. Microsoft is aware of the issue and has escalated it to their engineering team. We continue to work with their support to help investigate the issue and we will provide an update as soon as possible. In the interim, since workspace archivals may fail or complete without applying archival steps when Archival Policies include Microsoft 365 group visibility options (Remove from GAL / Remove from Exchange Client), as a workaround, temporarily disable these options and retry; no data loss has occurred.

4. monitoring Jan 14, 2026, 11:45 PM UTC

   Microsoft has confirmed this is a known, global issue affecting Microsoft Graph group updates and has begun deploying a fix. The fix is currently rolling out worldwide and is expected to be fully propagated within 2–3 business days; intermittent issues may continue during this period. As a workaround, disabling group visibility options (Remove from GAL / Remove from Exchange Client) remains recommended until the rollout is complete.

5. resolved Jan 20, 2026, 03:44 PM UTC

   On Thursday, January 15th we released a workaround in code to disable group visibility options (Remove from GAL / Remove from Exchange Client) on all calls to the Graph API. We have since tested this workaround and rolled it all to customers mitigating the impact of the issue. Microsoft continues to tell us that the issue has been patched by their Engineering team and that they have a fix being deployed, but they have given no timeline regarding when we can expect general resolution. We can confirm that the issue persists without the workaround mentioned above. Once we have confirmed that Microsoft has resolved the issue and that the group visibility options are again supported without error, we will reenable them. For the time being, we consider this incident resolved on our side.

6. postmortem Jan 29, 2026, 04:04 PM UTC

   ## Workspace Archival Failures \(Group Visibility Options\) ### Summary A Microsoft Graph service regression caused some workspace archivals to fail when Archival Policies included group visibility options \(**Remove from GAL** / **Remove from Exchange Client**\). In some cases, archivals appeared to succeed but did not apply any archival actions. ### Impact Only customers using the affected group visibility options were impacted. **No data loss occurred**, but impacted workspaces were not archived as expected until the issue was mitigated and archivals were re-run. ### Root Cause Microsoft Graph began returning errors when updating Microsoft 365 group visibility properties required by these archival options. This behavior changed unexpectedly and was external to Orchestry. ### Mitigation & Resolution Orchestry deployed a temporary workaround on **January 15** to disable the affected group visibility updates, which mitigated the issue for customers. On **Monday**, we confirmed Microsoft had fully resolved the underlying issue and removed the workaround. The archival options are now confirmed working normally.

Read the full incident report →

Started Oct 29, 2025, 04:00 PM UTC· resolved Oct 30, 2025, 12:35 AM UTC

1. investigating Oct 29, 2025, 04:00 PM UTC

   We are aware of a global issue affecting Azure that may cause intermittent access issues or degraded performance for some users. The problem originates with Microsoft’s infrastructure and is impacting multiple regions. Our team is monitoring the situation closely and will provide updates as Microsoft releases more information.

2. investigating Oct 29, 2025, 04:40 PM UTC

   We are continuing to monitor this issue for updates from Microsoft. Microsoft's latest update states that, "Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions." However, the incident appears to be affecting Azure Front Door in addition to Azure Portal. The outage is wide spread affecting regions globally. We will continue to update our status as our team learns more.

3. identified Oct 29, 2025, 05:20 PM UTC

   The latest update from Microsoft confirms that it is DNS affecting Azure Front Door and that they are rolling back recent changes to resolve the issue, "We are taking two concurrent actions where we are blocking all changes to the AFD services and disabling a problematic route that we found to be related to this, and at the same time rolling back to our last known good state ... We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update." We will update our status when we have confirmed that Microsoft's rollback has resolved the issue. In the meantime we continue to monitor.

4. monitoring Oct 29, 2025, 09:08 PM UTC

   Microsoft confirmed an Azure Front Door issue caused by a configuration change that led to latency and errors across multiple Azure services. They’ve rolled back the change and are rerouting traffic through healthy nodes, with recovery ongoing. Note that as recovery progresses, some requests may still land on unhealthy nodes, resulting in intermittent failures or reduced availability until more nodes are fully restored. Our services are stabilizing and performing normally again. We’ll continue to monitor until Microsoft confirms full recovery.

5. resolved Oct 30, 2025, 12:35 AM UTC

   Microsoft has confirmed that the Azure Front Door issues have been resolved. Orchestry services are fully functional.

6. postmortem Oct 30, 2025, 02:29 PM UTC

   ## Summary On October 29, 2025, Microsoft experienced a global issue affecting its Azure Front Door service, which handles traffic routing and content delivery for many cloud applications worldwide. This disruption caused intermittent slowdowns and brief unavailability across parts of the Orchestry platform. ## Impact Some users may have noticed temporary difficulty accessing Orchestry services or experienced slower page loads during the outage window. There was no impact to data integrity, security, or customer configurations, and all services continued to operate normally once Microsoft resolved the issue. ## Resolution Microsoft completed a rollback and restored full functionality to Azure Front Door by 00:40 UTC on October 30. Orchestry systems recovered at that time. ## Next Steps We continue to enhance our monitoring and resiliency measures to further minimize any future impact from upstream provider incidents. We appreciate your patience and understanding during this event.

Read the full incident report →

Started Nov 25, 2024, 10:30 AM UTC· resolved Nov 26, 2024, 01:00 PM UTC

1. identified Nov 25, 2024, 06:06 PM UTC

   Microsoft services (Exchange Online, Microsoft 365 suite, Microsoft Power Automate in Microsoft 365, Microsoft Purview, Microsoft Teams, SharePoint Online, Universal Print) are experiencing an active service degradation. As such, Orchestry services that depend on these services may fail to work. We are actively monitoring this situation. Microsoft has this Issue as ID MO941162 within Microsoft Administration Service Health.

2. monitoring Nov 26, 2024, 01:10 PM UTC

   On Nov 25th at 11:32 PM EST, Microsoft announced that core services have been restored, with the exception of Outlook on the web. We have confirmed that Orchestry services are no longer impacted.

3. resolved Nov 26, 2024, 01:27 PM UTC

   This incident has been resolved.

Read the full incident report →