FloQast Outage History

**Summary:** Customers using Reconciliations experienced an issue where changes made via the All Entities template were saved in settings but failed to apply to individual periods. This resulted in template updates not being reflected across the application and a lack of email notifications for these bulk updates. **Root Cause:** A standard infrastructure upgrade changed how internal system information was passed between background services. This caused a critical communication failure that prevented the automated workflow from distributing template changes to the appropriate records, which occurred without surfacing an error message to the user. **Immediate Resolution:** Engineering teams implemented a multi-layered fix that restored the necessary system context and updated the underlying software dependencies. After rigorous testing across multiple environments to ensure full functionality and the restoration of automated notifications, the fix was deployed to all customers. **Follow-up Actions:** To prevent a recurrence, the team is enhancing end-to-end regression testing for infrastructure changes and improving monitoring to ensure visibility into background service health. Additionally, system alerts are being updated to provide immediate notification of failures in automated workflows, and internal deployment checklists are being refined to ensure consistent dependency management during platform upgrades. **Recommended Customer Action: Any customers who made Reconciliation updates via the All Entities Reconciliation Template between 2:26 PM PT on April 20 and 3:16 PM PT on April 23 should re-apply their reconciliation templates to ensure the changes are properly applied.**

**Summary:** Clients experienced failures when attempting to push files through FloQast's SFTP integration. Automated file processing was interrupted across all production regions. **Root Cause:** A recent update to FloQast's file processing pipeline introduced a dependency on a new configuration that had been validated in a test environment but had not yet been applied to production. When the updated code was deployed to production, the underlying configuration it required was not in place, causing the file routing service to fail on startup. **Immediate Resolution:** Engineering teams identified the missing configuration and applied it across all production regions. **Follow-up Actions:** The team is implementing cross-repository deployment safeguards to ensure that when code changes and their corresponding infrastructure configurations are developed together, both are verified as deployed to production before either is considered complete. This will prevent scenarios where tightly coupled changes across different parts of the system can become inadvertently separated during the release process.

**Summary:** Clients experienced “Something Went Wrong” errors throughout certain parts of the application. **Root Cause:** As part of ongoing security infrastructure improvements, FloQast migrated services to a more robust web application firewall \(WAF\) configuration. In pre-production environments, a supplemental ruleset had been in place that, by design, took precedence over certain standard security rules — effectively masking how those rules would behave in production. When the migration to the stricter production security configuration occurred, legitimate integration traffic was inadvertently caught and blocked. **Immediate Resolution:** Engineering teams performed a full rollback of the affected deployments, restoring integration functionality for impacted customers. Once stable, the team began a careful analysis of the specific security rules to confirm that the appropriate level of restrictions was being adhered to. **Follow-up Actions:** The team is taking a deliberate approach to reintroducing the production security ruleset in monitoring mode before enforcing blocks, allowing for validation against known traffic patterns.

**Summary:** Clients experienced authorization errors \(403s\) when attempting to push data through FloQast's external API integration pipeline. **Root Cause:** As part of ongoing security infrastructure improvements, FloQast migrated services to a more robust web application firewall \(WAF\) configuration. In pre-production environments, a supplemental ruleset had been in place that, by design, took precedence over certain standard security rules — effectively masking how those rules would behave in production. When the migration to the stricter production security configuration occurred, legitimate integration traffic was inadvertently caught and blocked. **Immediate Resolution:** Engineering teams performed a full rollback of the affected deployments, restoring integration functionality for impacted customers. Once stable, the team began a careful analysis of the specific security rules to confirm that the appropriate level of restrictions was being adhered to. **Follow-up Actions:** The team is taking a deliberate approach to reintroducing the production security ruleset in monitoring mode before enforcing blocks, allowing for validation against known traffic patterns.

**Summary:** Clients experienced failures when attempting to load most Admin pages within the application. The Team page remained accessible, but all other Admin pages were unavailable during the incident window, which lasted approximately 27 minutes. **Root Cause:** During a scheduled release, certain application components were deployed out of the intended order. Specifically, updated client code was deployed before the underlying services it depended on were in place, causing Admin pages to fail to load when users attempted to access them. **Immediate Resolution:** Engineering teams identified the deployment sequencing gap and completed the deployment of the missing components. Once the remaining services were deployed in the correct order, Admin page functionality was fully restored across all affected users. **Follow-up Actions:** To prevent similar occurrences, the team is implementing clearer documentation of deployment ordering requirements for complex releases, ensuring all dependencies are explicitly captured and confirmed with relevant stakeholders before deployments begin. Internal release coordination processes are also being reviewed to reduce the risk of sequencing gaps, particularly in releases involving multiple simultaneous updates.

**Summary:** Clients experienced failures when attempting to load the Transform module, receiving an error indicating that their user identity or team membership could not be validated. The issue affected all users attempting to access Transform during the incident window. **Root Cause:** A code update introduced a dependency on a new policy definition that had not yet been deployed to the supporting authentication service. When users attempted to load Transform, the system was unable to complete the required identity and access validation, resulting in a load failure. **Immediate Resolution:** Engineering teams quickly identified that the issue stemmed from a mismatch between the recently deployed code and the underlying authentication service. The update was reverted to the previous stable release, which did not rely on the unavailable policy. This restored normal Transform functionality for all users within approximately 22 minutes of the issue being detected. **Follow-up Actions:** To prevent similar occurrences, the team is reinforcing the practice of ensuring all service dependencies are deployed and verified before releasing dependent code changes. Additionally, efforts are underway to improve how the application handles upstream service unavailability, with the goal of providing clearer, more actionable messaging to users rather than a hard failure, and to better equip support teams for faster diagnosis.