Mercatus Outage History

Started Apr 29, 2026, 07:11 AM UTC· resolved Apr 30, 2026, 12:33 AM UTC

1. investigating Apr 29, 2026, 07:11 AM UTC

   We are currently observing intermittent update failures in both Production and UAT affecting a subset of data updates backed by a lookup/persistence mechanism. There have been no recent configuration or deployment changes from our side, and the same workflows have been operating successfully for an extended period. At the same time, many update operations continue to complete as expected, indicating a potential underlying platform‑level constraint or behavioral limitation. We have reached out to our third‑party service provider to confirm whether any recent changes or limit adjustments on their side could be contributing to this behavior. In parallel, we are investigating: - Whether there is any ongoing service degradation or instability - Whether recent changes to lookup or persistence limits could be contributing to this behavior As a precautionary measure, we are also evaluating a move to a more scalable third‑party persistence solution to ensure long‑term reliability and consistency of updates. Further updates will be shared as more information becomes available. Thanks, CRPM Support Team

2. investigating Apr 29, 2026, 02:36 PM UTC

   Root Cause: As data volumes scaled over time, certain lookup‑based persistence mechanisms reached practical capacity thresholds, leading to intermittent update failures for a subset of workflows. This behavior surfaced gradually and only under specific data growth patterns, which is why many jobs continue to run successfully while others are impacted. Impact: A defined set of workflows in UAT (with similar exposure identified in Production) are affected. Data ingestion for the impacted workflows is currently paused to prevent inconsistent updates Current Actions: We are actively working with our third‑party service provider to validate system behavior and confirm whether any recent platform changes or capacity characteristics could be contributing to this pattern. In parallel, we are migrating affected workflows to a higher‑capacity persistence design better suited for sustained data growth and long‑term scalability Given the number of impacted workflows, this migration will be executed in a phased manner to ensure stability and controlled rollout Status: Remediation is in progress. We will share a more detailed timeline once the alternative persistence approach has been fully validated.

3. resolved Apr 30, 2026, 12:33 AM UTC

   The previously reported intermittent update failures have been resolved. The affected lookup‑based persistence has been migrated to a higher‑capacity data storage approach in both Production and UAT. This includes the reference cache and related staging structures used across ingestion workflows. Given the breadth of this change across ingestion processes, we are closely monitoring runs to ensure continued stability and data consistency. Please contact support if you observe any unexpected behavior.

Read the full incident report →

Started Jul 12, 2025, 09:03 AM UTC· resolved Jul 13, 2025, 02:34 AM UTC

1. investigating Jul 12, 2025, 09:03 AM UTC

   We are currently experiencing issues with platform integrations, resulting in failed or delayed integration runs. Our integration team began investigating the issue and is actively working to identify the root cause and implement a fix. We will share updates as soon as more information becomes available. We appreciate your patience and understanding.

2. identified Jul 12, 2025, 09:54 AM UTC

   🔄 Update: Integration Failures Status: Identified Our team has identified a connection issue with the database that is impacting integration functionality. The DevOps team is actively investigating the root cause. As a precaution, we have temporarily paused the polling recipes to prevent further failures. The estimated time for resolution is 2 PM ET (July 12, 2025). We will continue to provide updates as we work toward restoring full functionality. Thank you for your continued patience.

3. resolved Jul 13, 2025, 02:34 AM UTC

   The connection issue with the database has been successfully resolved. All integration services are now functioning as expected, and the polling recipes have been re-enabled. We have verified system stability and are continuing to monitor performance to ensure everything remains operational. Thank you for your patience and understanding throughout this incident.

4. postmortem Aug 05, 2025, 10:57 AM UTC

   ### 🔍 Summary On July 12, 2025, a product patch release triggered widespread failures across platform integrations, specifically affecting recipes and SFTP processes. The issue stemmed from a database connection failure and unintended execution of SFTP recipes, leading to queue congestion and delayed data ingestion. ### 📌 Root Cause * A **product patch** released on July 12 caused: * Failures in all recipes directly connected to the database. * Unintended execution of SFTP recipes, which processed all available files from Client’s MA SFTP servers simultaneously. * This led to **queue congestion** and disrupted integration workflows. ### 📊 Impact Summary **Affected Systems:** * Integration recipes \(DB connectors\) * SFTP integrations with client **Impacted Clients:** * Clients using integration services **Business Impact:** * Delayed and out-of-order data ingestion. * Manual intervention required to reset queues. * Risk of inaccurate downstream reporting and analytics. ### ⚠️ Contributing Factors * **No file archiving** during SFTP server migration. * **OS upgrade** from AWS Linux 2 to AWS Linux 2023 introduced a new network interface \(`ens5` instead of `eth0`\). * Existing NAT masquerading scripts were hardcoded to `eth0`, causing failures until updated. ### 🛠️ Mitigation & Resolution * **Immediate Actions:** * Paused polling recipes to prevent further failures. * DevOps team initiated investigation and identified the database connection issue. * **Resolution:** * Database connection issue resolved. * Polling recipes re-enabled. * System stability verified and performance monitoring initiated. ### ✅ Follow-Up Actions * Review and update all scripts to avoid hardcoded network interface dependencies. * Implement file archiving protocols during SFTP migrations. * Enhance patch testing procedures to include integration impact assessments. * Improve alerting and monitoring for recipe execution anomalies.

Read the full incident report →

Started Oct 29, 2024, 04:18 PM UTC· resolved Oct 29, 2024, 04:18 PM UTC

1. resolved Oct 29, 2024, 04:18 PM UTC

   On Monday 10/28 we experienced a PowerBI outage that began at 11:15 ET The outage impacted multiple regions, with some Clients experiencing the inability to refresh data due to regional PowerBI gateways going offline We quickly resolved the issue by 13:55 ET by working closely with Microsoft We are in the process of doing a root cause analysis in partnership with Microsoft and remediation to prevent additional occurrences. Additional details to follow.

Read the full incident report →