Is Permutive down?

We are currently investigating an issue affecting Audience and Campaign Insights, where some customers may see errors or missing data when loading Insights. This appears to be caused by heavy load on our reporting infrastructure, which is causing some queries to time out. This affects the Insights reporting layer only — data collection, cohort building, and campaign activation and delivery are not impacted, and no data is being lost. Our engineering team is actively working on it. We will provide a further update as soon as possible.

We have applied fixes to our reporting infrastructure, and Audience and Campaign Insights have recovered — queries are now loading normally in our testing. Earlier today, some customers experienced errors and missing data in Insights. This was caused by a temporary surge in processing: as data backfilling was carried out, the sudden influx of data generated unusually heavy queries, which triggered the reporting system's built-in safeguards and caused some Insights queries to time out. This affected the Insights reporting layer only — data collection, cohort building, and campaign activation and delivery were not impacted, and no data was lost. We are monitoring closely to confirm stability and will carry out a full review to prevent recurrence.

This incident has been resolved. Audience and Campaign Insights are fully operational and performing normally. Summary: On the afternoon of 4 June, some customers experienced errors and missing data when loading Audience and Campaign Insights over a period of roughly a few hours. The issue was caused by a surge of unusually heavy reporting queries — combined with a backfill of data — which triggered the reporting system's built-in load safeguards and caused some Insights queries to time out. This affected the Insights reporting layer only. Data collection, cohort building, and campaign activation and delivery were not impacted, and no data was lost. Resolution: We applied fixes to our reporting infrastructure on the evening of 4 June, which restored normal performance, and we have confirmed Insights is loading correctly across customers. Next steps: Our investigation identified inefficiencies in how some reporting data is indexed, which made certain queries more expensive than expected. We will be making improvements to the reporting system's data structure and storage over the coming weeks to make it more resilient and prevent recurrence. We apologise for the disruption and thank you for your patience.