Permutive Outage History

Started Sep 21, 2026, 09:59 AM UTC· resolved Sep 22, 2026, 07:57 PM UTC

1. identified Sep 21, 2026, 09:59 AM UTC

   Since 19 September 2026, the data shown in Insights (Audience & Campaign), Live Audience Stats and Plans has been delayed. Reporting in these products currently stops at 19 September and does not yet include the most recent days. The delay is caused by an issue with a cloud service we rely on to process reporting data. Our engineering team is working with the provider to resolve it. No data has been lost. Your events continue to be collected as normal, and the missing days will appear automatically once processing resumes. Audience targeting, activation and all other Permutive features are unaffected. We will post a further update as soon as possible. We apologise for the inconvenience.

2. monitoring Sep 21, 2026, 03:18 PM UTC

   The underlying issue has been resolved, and data ingestion has resumed. We are now loading the delayed data from 19 September onwards, and it will progressively appear in Insights, Live Audience Stats and Plans as it is processed. No data has been lost. We will continue monitoring and post a further update once reporting is fully up to date.

3. monitoring Sep 22, 2026, 12:52 PM UTC

   Recovery is nearly complete. The majority of the delayed data from 19 September onwards is now visible in Insights, Live Audience Stats and Plans, and the final portion is being loaded. Some customers may still see a small gap for the most recent days until this finishes. No data has been lost. We will post a final update once we have confirmed that reporting is fully up to date for all customers.

4. resolved Sep 22, 2026, 07:57 PM UTC

   This incident has been fully resolved. All delayed data from 19 September onwards has been loaded, and reporting in Insights, Live Audience Stats and Plans is now up to date. No data was lost during this period. The delay was caused by a change to a cloud service we use to process reporting data. We have since changed how our pipelines use that service so that this type of change can no longer disrupt processing. Thank you for your patience.

Read the full incident report →

Started Sep 03, 2026, 08:00 PM UTC· resolved Sep 03, 2026, 10:13 PM UTC

1. resolved Sep 04, 2026, 03:38 PM UTC

   Between 21:14 and 22:13 UTC on 3 September, the CCS (Custom Cohort Segmentation) API experienced degraded performance. During this period around a third of segmentation requests, and up to half at peak, returned HTTP 500 errors after a delay of several seconds. Events sent in the affected requests were still collected. However, no cohorts were returned for those requests, and the user's server-side segmentation state was not updated with those events This only affected segmentation done via direct calls to the CCS API. Requests outside this window were unaffected. The issue is now resolved and the API has returned to normal. This was caused by a failure in an upstream service which impacted the CCS API. Improvements to both the upstream service and the CCS API have been identified and are being worked on which will prevent a reoccurrence of this failure and its impact.

Read the full incident report →

Started Sep 01, 2026, 02:27 PM UTC· resolved Sep 01, 2026, 02:43 PM UTC

1. investigating Sep 01, 2026, 02:27 PM UTC

   We are investigating an outage of Permutive API.

2. monitoring Sep 01, 2026, 02:35 PM UTC

   A fix has been implemented and we're continuing to monitor.

3. resolved Sep 01, 2026, 02:43 PM UTC

   This incident is now resolved. We will update with more information soon.

Read the full incident report →

Started Jun 04, 2026, 04:32 PM UTC· resolved Jun 05, 2026, 10:16 AM UTC

1. investigating Jun 04, 2026, 04:32 PM UTC

   We are currently investigating an issue affecting Audience and Campaign Insights, where some customers may see errors or missing data when loading Insights. This appears to be caused by heavy load on our reporting infrastructure, which is causing some queries to time out. This affects the Insights reporting layer only — data collection, cohort building, and campaign activation and delivery are not impacted, and no data is being lost. Our engineering team is actively working on it. We will provide a further update as soon as possible.

2. monitoring Jun 04, 2026, 07:21 PM UTC

   We have applied fixes to our reporting infrastructure, and Audience and Campaign Insights have recovered — queries are now loading normally in our testing. Earlier today, some customers experienced errors and missing data in Insights. This was caused by a temporary surge in processing: as data backfilling was carried out, the sudden influx of data generated unusually heavy queries, which triggered the reporting system's built-in safeguards and caused some Insights queries to time out. This affected the Insights reporting layer only — data collection, cohort building, and campaign activation and delivery were not impacted, and no data was lost. We are monitoring closely to confirm stability and will carry out a full review to prevent recurrence.

3. resolved Jun 05, 2026, 10:16 AM UTC

   This incident has been resolved. Audience and Campaign Insights are fully operational and performing normally. Summary: On the afternoon of 4 June, some customers experienced errors and missing data when loading Audience and Campaign Insights over a period of roughly a few hours. The issue was caused by a surge of unusually heavy reporting queries — combined with a backfill of data — which triggered the reporting system's built-in load safeguards and caused some Insights queries to time out. This affected the Insights reporting layer only. Data collection, cohort building, and campaign activation and delivery were not impacted, and no data was lost. Resolution: We applied fixes to our reporting infrastructure on the evening of 4 June, which restored normal performance, and we have confirmed Insights is loading correctly across customers. Next steps: Our investigation identified inefficiencies in how some reporting data is indexed, which made certain queries more expensive than expected. We will be making improvements to the reporting system's data structure and storage over the coming weeks to make it more resilient and prevent recurrence. We apologise for the disruption and thank you for your patience.

Read the full incident report →

Started Apr 17, 2026, 01:21 PM UTC· resolved Apr 20, 2026, 04:34 PM UTC

1. monitoring Apr 17, 2026, 01:21 PM UTC

   Between 26 March 2026 and 17 April 2026, segment metadata was not being delivered to streaming routing BigQuery destinations. This was identified on 15th April. All other event routing continued to operate normally throughout this period and was unaffected. We have identified the root cause, and a fix has already been rolled out to the affected destinations. A tested backfill solution for segment metadata is available. Please contact [email protected] for more details. We sincerely apologise for the disruption and for the delay in communicating it. As a follow-up, we are investing in improved monitoring so that per-destination write failures are detected proactively in future.

2. resolved Apr 20, 2026, 04:34 PM UTC

   The fix has been fully rolled out, and segment metadata delivery to streaming Routing BigQuery destinations has been confirmed as restored. All affected destinations are operating normally. A tested backfill solution for segment metadata remains available — please contact [email protected] if you would like us to restore segment metadata for the affected period. Thank you for your patience throughout this incident.

Read the full incident report →

Started Apr 07, 2026, 02:30 PM UTC· resolved Apr 08, 2026, 03:41 PM UTC

1. investigating Apr 07, 2026, 02:30 PM UTC

   We are aware of an issue affecting audience volume data displayed in Insights dashboards. Some customers may notice reduced audience sizes for cohorts since 4 April. Underlying data collection is unaffected and no data has been lost. The issue is limited to how volumes are reflected in the dashboard. Our engineering team is actively working on a fix and we expect to recover the affected data. We will provide a further update as soon as we have more information.

2. identified Apr 07, 2026, 04:23 PM UTC

   We have identified a contributing cause and recovery is underway. Data in Audience Insights and Campaign Insights has been restored. Plans data repair is still in progress.

3. resolved Apr 08, 2026, 03:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 03, 2026, 09:47 AM UTC· resolved Mar 03, 2026, 07:01 PM UTC

1. identified Mar 03, 2026, 09:47 AM UTC

   We are currently experiencing issues with our Insights platform, which is impacting the availability of the following areas: Planning Cohort Sizes AI Recommendations (Optimisation) Dashboard Our Engineering team has identified the issue and is actively working to restore full service.

2. monitoring Mar 03, 2026, 01:52 PM UTC

   We are continuing to work on restoring the affected Insights services. Our team has identified the issue and has implemented changes to resolve it. All impacted services are currently recovering and returning to fully operational: - Dashboard - Planning - AI Recommendations in Optimization - Live Audience Sizes - Cohort Export We will continue to monitor the situation and provide a further update once we have confirmed the issue has been resolved.

3. resolved Mar 03, 2026, 07:01 PM UTC

   This incident has been resolved and all impacted services are fully operational: Planning Cohort Sizes AI Recommendations (Optimisation) Dashboard

Read the full incident report →

Started Feb 25, 2026, 03:55 PM UTC· resolved Mar 02, 2026, 09:00 AM UTC

1. identified Feb 25, 2026, 03:55 PM UTC

   We are currently experiencing infrastructure issues causing delays to the following services: - Audience Imports Ingestion – Delays in data ingestion - Batch Routing – Delays in batch data delivery (stream routing remains fully operational) - Insights – Delays in data availability - Event Metrics in Dashboard – Delays in metrics updates - Server-Side Activations – Delays in segment activation data being sent to partners Client-side segmentation and activation continue to operate normally. Our engineering team is investigating and we will provide updates as the situation develops.

2. identified Feb 26, 2026, 08:26 AM UTC

   We are continuing to work on a fix for this issue.

3. identified Feb 26, 2026, 08:27 AM UTC

   Server-Side Activations are now fully operational. We are continuing to work on restoring the remaining affected services: Audience Imports Ingestion – Delays in data ingestion Batch Routing – Delays in batch data delivery (stream routing remains fully operational) Insights – Delays in data availability Event Metrics in Dashboard – Delays in metrics updates Client-side segmentation and activation continue to operate normally. We will provide further updates as the situation progresses.

4. identified Feb 27, 2026, 02:28 PM UTC

   The following services are now fully operational: Server-Side Activations Insights Audience Imports Ingestion Batch Routing We are continuing to work on restoring Event Metrics in Dashboard. Client-side segmentation and activation continue to operate normally. We will provide further updates as the situation progresses.

5. identified Mar 02, 2026, 08:59 AM UTC

   We are continuing to work on a fix for this issue.

6. resolved Mar 02, 2026, 09:00 AM UTC

   All affected services are now fully operational: Audience Imports Ingestion Batch Routing Insights Event Metrics in Dashboard Server-Side Activations Client-side segmentation and activation operated normally throughout the incident. No data was lost. We appreciate your patience.

Read the full incident report →

Started Feb 24, 2026, 07:22 PM UTC· resolved Feb 24, 2026, 10:34 PM UTC

1. identified Feb 24, 2026, 07:22 PM UTC

   We are currently experiencing a delay to data delivery for Routing. Our Engineering team are investigating this issue.

2. resolved Feb 24, 2026, 10:34 PM UTC

   This issue has been resolved, with data delivery catching up at circa 22:20 UTC.

Read the full incident report →

Started Feb 18, 2026, 11:31 AM UTC· resolved Feb 18, 2026, 05:23 PM UTC

1. identified Feb 18, 2026, 11:31 AM UTC

   We have identified an issue affecting our Routing product where some events are experiencing delayed delivery to customer data warehouses. While the majority of events are being delivered promptly, a subset may be delayed by up to 6 hours. The cause has been identified and a fix is currently being applied. We will provide a further update once delivery times have returned to normal. We apologise for any inconvenience.

2. identified Feb 18, 2026, 04:55 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Feb 18, 2026, 05:23 PM UTC

   The issue causing delayed event delivery in Routing has been resolved. All events have been delivered and delivery times have returned to normal. No events were lost during the incident. We apologise for the disruption.

Read the full incident report →

Started Jan 19, 2026, 07:50 AM UTC· resolved Jan 23, 2026, 10:09 PM UTC

1. identified Jan 19, 2026, 07:50 AM UTC

   Permutive Routing is currently experiencing severe delays. Customers who use Routing to sync data to their cloud data lake or warehouse will have delays in data delivery since 18th January 9:46 PM UTC. Relatedly, customers will be experiencing delays in third-party data imports. We are working to process all data and restore real-time delivery.

2. identified Jan 19, 2026, 03:45 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Jan 19, 2026, 04:46 PM UTC

   Our engineering team has identified the root cause as certain infrastructure components reaching capacity limits. We have brought additional infrastructure online and are actively migrating data to distribute the load more effectively. We are seeing improvements on some affected systems and expect continued progress as migrations complete. We will provide further updates as the situation develops. We appreciate your patience and apologise for the ongoing disruption.

4. identified Jan 20, 2026, 05:44 PM UTC

   We are continuing to work on resolving this issue. Our engineering team has taken additional steps to accelerate infrastructure recovery: - Deployed changes to redirect traffic away from the impacted infrastructure - Initiated partition leader reassignment across the cluster (currently ~50% complete) - Applied replication throttling to reduce load on affected nodes Affected systems and current impact: - Routing - Severe delays in data delivery to cloud data lakes/warehouses - Third-Party Data & Audience Imports - Delays in data ingestion and processing - Live Audience Size (LAS) & Insights - Unreliable numbers in the dashboard due to ingestion delays, including event counts and latest events samples - Server-Side Integrations - Delays in segment activation data being sent to partners No data has been lost during this incident. All data is being queued and will be processed once systems have recovered. We are working to restore normal processing as quickly as possible. We will provide further updates as the situation develops.

5. monitoring Jan 21, 2026, 11:26 AM UTC

   We have made significant progress overnight. Infrastructure load has reduced significantly, and under-replicated data is back in sync. Routing is now fully back online and processing its backlog. We are bringing the remaining services back online in a phased approach throughout the day, monitoring system health at each stage. Topic retention has been extended to ensure no data loss during recovery. We will provide further updates as recovery progresses.

6. monitoring Jan 22, 2026, 03:26 PM UTC

   We are continuing to work on resolving this issue. Our engineering team is performing additional infrastructure work to improve stability. To allow the infrastructure to stabilise, some services have been temporarily scaled down again. This is expected and part of the recovery process. Current impact: - Routing - A large number of integrations have caught up, but ingestion has been paused again; data delays expected - Third-Party Data & Audience Imports - Delays in data ingestion; some processing paused - Audience & Campaign Insights - Available and fully operational - Audience Planning Insights, LAS & AI Recommendations - Experiencing some volatility in performance. Some features may be temporarily unavailable while the platform stabilises - Server-Side Integrations - Delays in segment activation data being sent to partners We will continue to provide updates as the situation progresses.

7. resolved Jan 23, 2026, 10:09 PM UTC

   This incident is now resolved, and normal operations have resumed across all affected services: — Routing: Fully operational; all data syncing to cloud data lakes/warehouses in real-time — Third-Party Data & Audience Imports: Fully operational — Audience & Campaign Insights: Fully operational — Audience Planning Insights, Live Audience Sizes, and AI Recommendations: Fully operational — Server-Side Integrations: Fully operational Most queued data has been successfully processed, with some remaining queued data to be processed over the next few hours. No data was lost during this incident, but we appreciate that customers experienced delayed data delivery for some of the impacted services. Client-side segmentation and activation continued to operate fully throughout the incident. We will continue to monitor this infrastructure closely over the coming days as a precaution. We appreciate your patience throughout this incident.

Read the full incident report →

Started Dec 05, 2025, 09:13 AM UTC· resolved Dec 05, 2025, 09:32 AM UTC

1. identified Dec 05, 2025, 09:13 AM UTC

   We are currently experiencing an issue with our CDN provider, which began at 08:48 UTC. Our CDN provider delivers the Permutive SDK to end-users on Web environments and is also used to serve the Permutive Dashboard. For new users that do not have the SDK cached, event capture and targeting will not function as expected. Customers currently cannot access the Permutive Dashboard. We are working with Cloudflare to resolve this issue as soon as possible.

2. resolved Dec 05, 2025, 09:32 AM UTC

   Our CDN provider's outage was resolved at 09:11 UTC, restoring full service to the Permutive Dashboard and SDKs. We will continue to monitor this throughout the day.

Read the full incident report →

Started Dec 12, 2024, 10:34 AM UTC· resolved Dec 12, 2024, 11:51 AM UTC

1. resolved Dec 13, 2024, 01:20 PM UTC

   Permutive Events API experienced an outage. This partial outage started at 11:34:44 UTC and was resolved at 11:51:30 UTC For affected customers, segmentation & activation of users remained operational with degraded service. Insights and Routing were not operational due to this issue.

Read the full incident report →