Piano.io Outage History
Piano.io is up right nowPiano.io had 51 outages in the last 2 years totaling 843h 11m of downtime — averaging 2.1 incidents per month.
There were 51 Piano.io outages since June 18, 2025 totaling 843h 11m of downtime. Each is summarised below — incident details, duration, and resolution information.
Content Crawler Service Disruption
Affected: Piano Content Crawler Data Collection
Timeline · 6 updates
- investigating May 29, 2026, 02:31 AM UTC
We are currently experiencing an issue with the Content Crawler, which is failing to crawl content. Our team is actively investigating the root cause. We will provide further updates as more information becomes available.
- investigating May 29, 2026, 02:31 AM UTC
We are continuing to investigate this issue.
- identified May 29, 2026, 08:39 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 29, 2026, 09:24 AM UTC
A fix has been implemented and all Content pipeline services have been restored. Some processing delays may persist while the crawling backlog clears. We will continue to monitor and confirm full resolution.
- resolved May 29, 2026, 02:03 PM UTC
This incident has been resolved.
- postmortem Jun 04, 2026, 09:00 AM UTC
RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/06/2026-05-29-RCA-Content-Crawler-Service-Disruption-1.pdf).
Audience - API Failures happening in Europe environment.
Affected: API Endpoints - cxense - api.cxense.com
Timeline · 4 updates
- identified May 28, 2026, 04:27 PM UTC
We identified an issue in our systems starting around 10:20 UTC that is currently degrading CXSense platform responsiveness over the /public/user/id endpoint. As a result, some users may experience failed requests, while others may experience longer response times and timeouts. Our engineering team is actively investigating and working on mitigation efforts. We will continue to provide updates as more information becomes available.
- monitoring May 28, 2026, 05:10 PM UTC
A fix has been implemented and we are currently monitoring the results. At this time, API performance has returned to normal operation levels and the platform is behaving as expected again. We will continue monitoring closely to ensure stability.
- resolved May 29, 2026, 03:53 AM UTC
This incident has been resolved.
- postmortem Jun 15, 2026, 09:25 AM UTC
RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/06/2026-05-28-RCA-Audience-API-Failures-happening-in-Europe-environment.pdf).
[Activation] Service disruption (US region)
Affected: Publisher Dashboard - US - https://dashboard.piano.ioVX Checkout - US - buy.piano.ioAPI Endpoints - US - api.piano.ioUS - id.piano.io
Timeline · 2 updates
- monitoring May 20, 2026, 10:19 AM UTC
On 20 May 2026, starting at approximately 09:58 UTC, we identified an issue affecting Piano Activation in the US environment, impacting the Dashboard, Piano ID authentication, and checkout flows. A fix has been applied and the platform is now operating normally. We are continuing to monitor the situation and further details will follow. We apologise for any inconvenience caused.
- resolved May 20, 2026, 12:41 PM UTC
The incident has been resolved. All services are operating normally. Between approximately 09:58 and 10:11 UTC, users in the US region may have experienced issues accessing the Piano Dashboard, signing in via Piano ID, or completing checkout flows. This was caused by a routine configuration update to our US environment that introduced a conflict and disrupted traffic routing to the affected services. Our team identified the issue, rolled back the change, and confirmed full recovery shortly after. To prevent this from happening again, we are reviewing our release process and will validate the change in a controlled manner before re-applying it. We apologize for the inconvenience and thank you for your patience.
[Activation] Publisher Dashboard Degraded Performance (AP region)
Affected: Publisher Dashboard - Asia - https://dashboard-ap.piano.io
Timeline · 2 updates
- identified May 15, 2026, 08:03 AM UTC
We are currently experiencing degraded performance across the Publisher Dashboard in the AP environment, with data taking up to 3 hours longer than usual to reflect recent changes. This may affect any section of the Dashboard — including user search, reporting, and other data views. Our engineering team has identified the underlying cause and is actively working on a resolution. We will provide updates as the situation progresses.
- resolved May 16, 2026, 07:16 AM UTC
This incident has been resolved.
Europe - Composer - Experiences targeting custom fields are failing
Affected: Composer Experience Execution - Europe
Timeline · 4 updates
- investigating May 13, 2026, 01:48 PM UTC
We are currently investigating issues affecting the execution of experiences in our European region for segmentation that relies on custom field rules. Impacted users may notice that experiences are not triggering as expected, not reaching the desired outcome. Our engineering team is actively investigating the issue.
- monitoring May 13, 2026, 02:02 PM UTC
The issue has been identified and fixed; we're currently monitoring the experience executions.
- resolved May 13, 2026, 06:00 PM UTC
This incident has been resolved.
- postmortem May 15, 2026, 09:39 AM UTC
An infrastructure update to the EU environment introduced an incompatibility that prevented Composer from retrieving Custom Fields. The issue was limited to the EU region and did not affect other environments. The update introduced a behavioural change in the network routing layer where a configuration setting that had previously been relied upon was no longer honoured under the new setup. This incompatibility was not detected during pre-production testing, as the test environments did not fully replicate the conditions needed to surface the issue. The root cause was identified and the configuration gap was addressed promptly. Piano is taking steps to improve parity between pre-production and production environments and to strengthen the validation process ahead of future infrastructure changes to prevent similar issues.
[Activation] Elevated API Error Rates (EU region)
Affected: API Endpoints - EU - api-eu.piano.ioEU - id.piano.io
Timeline · 1 update
- resolved May 13, 2026, 08:52 AM UTC
Between 07:42 and 07:52 AM UTC, Piano experienced elevated error rates affecting the Publisher API and ID API in the EU environment. The incident occurred during an ingress upgrade from a deprecated version to the current supported version, which caused a routing configuration incompatibility. The issue lasted approximately 10 minutes and has been fully resolved. Publisher Dashboard, Checkout, and access controls in Composer operated normally throughout the incident and were not affected.
[Piano Analytics] Incident supported by the on-call team
Timeline · 2 updates
- investigating May 08, 2026, 10:21 PM UTC
We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.
- resolved May 08, 2026, 10:21 PM UTC
The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.
Composer C1X segmentation issues (EU & US regions)
Affected: API Endpoints - cxense - api.cxense.comOther
Timeline · 3 updates
- investigating May 08, 2026, 06:04 AM UTC
Composer is experiencing a service incident affecting C1X segmentation in both EU and US regions. Impact includes high error rates, degraded segment lookup, and degraded widget performance. The issue started at approximately 22:00 UTC on May 7 and continues with periods of instability. Our team is actively investigating the issue.
- monitoring May 08, 2026, 09:26 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 08, 2026, 05:36 PM UTC
The incident has been resolved.
Piano Sandbox - Composer currently not working
Timeline · 4 updates
- identified May 06, 2026, 01:17 PM UTC
We are currently experiencing an issue with Composer in our sandbox environment; the Composer module and experiences fail to load. Our team has identified the root cause and is actively working to resolve the issue. We will provide updates as more information becomes available.
- monitoring May 06, 2026, 01:43 PM UTC
The team has deployed a fix, and the issue has been resolved. Experiences are executing, and the Composer module is working normally. We will continue monitoring the remediation.
- monitoring May 06, 2026, 01:44 PM UTC
We are continuing to monitor for any further issues.
- resolved May 07, 2026, 06:28 AM UTC
This incident has been resolved.
[Activation] Increased API Error Rate on ID Endpoints (US region)
Affected: US - id.piano.io
Timeline · 2 updates
- monitoring May 05, 2026, 08:21 AM UTC
Between 06:48 and 07:05 UTC, Piano ID API experienced elevated error rates in the US cluster. Requests requiring user authentication, token verification, and user profile retrieval were affected. Other Piano services including Composer and VX subscription management were not impacted. Service has been fully restored and all systems are operating normally. We apologize for any disruption caused during this incident.
- resolved May 05, 2026, 08:21 AM UTC
This incident has been resolved.
Scheduled Maintenance: Ingress Controller Upgrade (US region)
Timeline · 1 update
- monitoring Apr 28, 2026, 04:00 AM UTC
THIS IS A SCHEDULED EVENT Apr 28, 04:00 - 05:00 UTC Apr 17, 15:44 UTC Scheduled - Piano is planning an infrastructure upgrade for the US region. This update is needed in order to modernize our ingress controller architecture by transitioning from ingress-nginx to F5 Nginx ingress. While we do not expect any impact, brief API disruptions (up to 15 minutes) may affect: Piano ESP API Endpoints Piano ESP Data Collection Piano ESP On-Site Recommendations Piano ESP Dashboard (US) Timings: April 28, 2026 | 04:00 - 05:00 UTC
Composer waterfall issues on US environment.
Affected: Composer Experience Execution - US
Timeline · 5 updates
- investigating Apr 27, 2026, 04:57 PM UTC
We have found an issue with our priority rules for API interaction on server/edge in Composer. The dev team is currently investigating. Clients may find their list order being changed or unsorted without any user interaction. We will continue updating this incident once we identify the root cause.
- investigating Apr 27, 2026, 06:37 PM UTC
Our team has restored the priority rules to their pre-incident state while we continue to investigate the root cause. We advise customers to avoid making changes to priority rules until the investigation is complete. We will provide further updates as more information becomes available.
- monitoring Apr 27, 2026, 10:01 PM UTC
The issue has been identified and mitigated; we will continue monitoring the remediation.
- resolved Apr 28, 2026, 09:20 AM UTC
This incident has been resolved.
- postmortem Apr 28, 2026, 02:04 PM UTC
RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2025-05-06-RCA-Experience-Priority-Data-Incident.pdf).
Elevated server latency in Composer execution
Affected: Composer Experience Execution - US
Timeline · 4 updates
- identified Apr 23, 2026, 02:35 PM UTC
Composer is experiencing a latency incident affecting executions in the US region starting at 10 AM UTC. Impact includes timeouts and degraded performance.
- monitoring Apr 23, 2026, 05:31 PM UTC
The incident has been resolved. We will continue monitoring the latency on the backend servers.
- monitoring Apr 23, 2026, 05:50 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 24, 2026, 10:42 AM UTC
This incident has been resolved.
[Activation] Publisher Dashboard - Composer Experience version selection impacted
Affected: Publisher Dashboard - US - https://dashboard.piano.ioPublisher Dashboard - Asia - https://dashboard-ap.piano.ioPublisher Dashboard - Australia - https://dashboard-au.piano.ioPublisher Dashboard - Europe - https://dashboard-eu.piano.io
Timeline · 3 updates
- investigating Apr 21, 2026, 09:00 AM UTC
We are currently investigating an issue affecting the publisher dashboard where users are unable to interact with the experience version dropdown and other buttons in Composer.
- identified Apr 21, 2026, 09:22 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Apr 21, 2026, 09:59 AM UTC
This incident has been resolved. The problem was caused by a compatibility issue between two system components that were running different versions. We have updated one of the components to ensure they work together properly, which has restored normal functionality. Some users may need to clear their browser cache to fully resolve the issue and restore normal functionality. We apologize for any disruption this issue may have caused.
[Activation] Composer segmentation issues (EU region)
Affected: API Endpoints - cxense - api.cxense.comOther
Timeline · 4 updates
- investigating Apr 15, 2026, 06:09 AM UTC
Composer is experiencing a service incident affecting Audience segments and Propensity Models in the EU region. Impact includes high error rates, degraded segment lookup, and degraded widget performance.
- identified Apr 15, 2026, 06:54 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 15, 2026, 08:45 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 17, 2026, 06:28 AM UTC
This incident has been resolved.
ID and VX Transaction Emails not being delivered due to Internal Mandrill Issue
Affected: Piano ID and VX transactional Emails
Timeline · 3 updates
- identified Apr 12, 2026, 05:00 AM UTC
Mandrill team has notified Piano about an issue impacting our account. Our team is currently coordinating efforts with the Mandrill account. The emails are queued but not getting delivered at the moment. We will continue to provide updates via this thread. Please monitor our status page for latest information.
- monitoring Apr 13, 2026, 03:32 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 13, 2026, 08:46 AM UTC
This incident has been resolved.
Scheduled Maintenance: Ingress Controller Upgrade (EU region)
Timeline · 1 update
- resolved Apr 16, 2026, 05:00 AM UTC
Apr 16, 05:00 UTC Completed - The scheduled maintenance has been completed. Apr 16, 04:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr 9, 08:16 UTC Scheduled - Piano is planning an infrastructure upgrade for the European region. This update is needed in order to modernize our ingress controller architecture by transitioning from ingress-nginx to F5 Nginx ingress. While we do not expect any impact, brief API disruptions (up to 15 minutes) may affect: Piano ESP API Endpoints Piano ESP Data Collection Piano ESP On-Site Recommendations Piano ESP Dashboard (EU) Timings: April 16, 2026 | 04:00 - 05:00 UTC
Scheduled Maintenance: Ingress Controller Upgrade (AP region)
Timeline · 1 update
- resolved Apr 15, 2026, 04:00 PM UTC
Apr 15, 16:00 UTC Completed - The scheduled maintenance has been completed. Apr 15, 15:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr 9, 08:14 UTC Scheduled - Piano is planning an infrastructure upgrade for the Asia Pacific region. This update is needed in order to modernize our ingress controller architecture by transitioning from ingress-nginx to F5 Nginx ingress. While we do not expect any impact, brief API disruptions (up to 15 minutes) may affect: Piano ESP API Endpoints Piano ESP Data Collection Piano ESP On-Site Recommendations Piano ESP Dashboard (AP) Timings: April 15, 2026 | 15:00 - 16:00 UTC
[Activation] Site Licensing incident (AP region)
Affected: Other
Timeline · 3 updates
- monitoring Apr 06, 2026, 01:53 PM UTC
Following an infrastructure update on April 3, a timezone-related bug was introduced, causing anomalies in Site Licensing, Subscription Grace Periods, and Scheduled Upgrades for clients in the Production-AP region. The root cause has been identified and a fix has been deployed. Remediation of the affected entities is in progress. A detailed Root Cause Analysis report will be shared once the incident is fully resolved, covering the scope of impact, timeline, and preventive measures.
- resolved Apr 06, 2026, 05:27 PM UTC
The incident has been resolved. A Root Cause Analysis report is currently in progress and will be shared once completed.
- postmortem Apr 28, 2026, 10:35 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2026-04-06-Activation-Timezone-Configuration-Error-AP-region.pdf).
[Activation] Subscription Log Performance Upgrade (US region)
Timeline · 1 update
- resolved Mar 31, 2026, 09:00 AM UTC
Mar 31, 09:00 UTC Completed - The scheduled maintenance has been completed. Mar 31, 07:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar 24, 17:11 UTC Scheduled - On Tuesday, March 31, 2026, between 07:00-09:00 UTC, Piano is planning a scheduled maintenance to improve the overall performance of the Subscription Log report for the US region. During this maintenance period, the report will continue to function normally; however, there will be some data lag, with the latest changes not being immediately reflected in the report. This temporary delay is expected to last for the duration of the maintenance window, up to 2 hours. Normal, real-time data updates will resume immediately after the maintenance is completed. We appreciate your understanding and patience as we work to enhance our services.
[Activation] Subscription Log Performance Upgrade (EU region)
Timeline · 1 update
- resolved Mar 30, 2026, 10:00 AM UTC
Mar 30, 10:00 UTC Completed - The scheduled maintenance has been completed. Mar 30, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar 24, 17:09 UTC Scheduled - On Monday, March 30, 2026, between 08:00-10:00 UTC, Piano is planning a scheduled maintenance to improve the overall performance of the Subscription Log report for the EU region. During this maintenance period, the report will continue to function normally; however, there will be some data lag, with the latest changes not being immediately reflected in the report. This temporary delay is expected to last for the duration of the maintenance window, up to 2 hours. Normal, real-time data updates will resume immediately after the maintenance is completed. We appreciate your understanding and patience as we work to enhance our services.
Login captcha feature not working correctly
Affected: US - id.piano.ioAustralia - id-au.piano.ioAsia - id-ap.piano.ioEU - id.piano.io
Timeline · 4 updates
- investigating Mar 19, 2026, 03:14 PM UTC
We've received some complaints that end users are unable to pass the CAPTCHA verification. Our development team is currently looking into the issue.
- identified Mar 19, 2026, 03:37 PM UTC
We have identified the issue, and our team is actively working on a fix. We'll keep you updated on our progress.
- monitoring Mar 19, 2026, 03:40 PM UTC
We have successfully implemented a fix for the CAPTCHA issue. We're actively monitoring the system.
- resolved Mar 19, 2026, 04:45 PM UTC
The incident has been resolved.
[Activation] Webhook Delivery Issue (US Region)
Affected: Other
Timeline · 5 updates
- identified Mar 06, 2026, 01:10 PM UTC
We’re currently experiencing an issue affecting webhook deliveries in the US region. Some webhooks may fail or contain invalid data. Our team has identified the cause and is actively working on a fix. Updates will follow as more information becomes available.
- identified Mar 06, 2026, 01:27 PM UTC
We have identified an issue causing corrupted webhook payloads for a subset of users. Due to a database limitation where a foreign key column hit its maximum value, webhooks may contain corrupted metadata fields, including malformed data like '*mac.com' instead of correct information. Our team is actively working on a fix, and we'll keep you updated on our progress.
- monitoring Mar 06, 2026, 01:43 PM UTC
We have successfully implemented a fix for the webhook payload issue. New events are now being delivered with correct content again. To expedite the resolution, events were rerouted to a new table, which means historical events from before 13:31 UTC are temporarily unavailable while we work on restoring them. We're actively monitoring the system and continuing restoration efforts, though our preliminary assessment indicates that data from the affected time period (11:20 UTC - 13:31 UTC) may not be completely recoverable.
- resolved Mar 06, 2026, 06:05 PM UTC
The issue has been resolved, and the team will continue working on restoring the missing events.
- postmortem Mar 30, 2026, 08:48 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/03/2026-03-06-RCA-Activation-Webhooks-incident.pdf).
Amplifier performance issues
Affected: SocialFlow.com Site uptimeSocialFlow publishing
Timeline · 2 updates
- resolved Feb 20, 2026, 10:12 AM UTC
On February 19, 2026 between 12pm and 10pm UTC, Piano experienced a production incident affecting Amplifier (Socialflow). The incident impacted compose box functionality for a subset of users (browser-dependent). Some users also experienced degraded UI performance. Additionally, publishing was affected for users who rely on the compose box for manual post publishing. The issue has been resolved by rolling back Socialflow Web to the previous release. We are currently investigating the root cause and will publish an incident report once the analysis is complete.
- postmortem Mar 12, 2026, 12:40 PM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/03/2026-02-19-RCA-Amplifier-degraded-performance.pdf).