Piano.io Outage History
Piano.io is up right nowPingoru has recorded 84 incidents for Piano.io: 80 in the last 12 months and 13 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 15 | 231 of 720 (32%) |
| May 2026 | 10 | 81 of 744 (11%) |
| June 2026 | 2 | 34 of 720 (5%) |
| July 2026 | 5 | 173 of 744 (23%) |
| August 2026 | 5 | 22 of 744 (3%) |
| September 2026 | 5 | 18 of 720 (2%) |
| October 2026 | 3 | 23 of 591 (4%) |
Incidents, newest first
Notice May 13, 2026 [Activation] Elevated API Error Rates (EU region) 10m API Endpoints - EU - api-eu.piano.io, Registration + Login - EU - id.piano.io
Started May 13, 2026, 07:42 AM UTC· resolved May 13, 2026, 07:52 AM UTC
- resolved May 13, 2026, 08:52 AM UTC
Between 07:42 and 07:52 AM UTC, Piano experienced elevated error rates affecting the Publisher API and ID API in the EU environment. The incident occurred during an ingress upgrade from a deprecated version to the current supported version, which caused a routing configuration incompatibility. The issue lasted approximately 10 minutes and has been fully resolved. Publisher Dashboard, Checkout, and access controls in Composer operated normally throughout the incident and were not affected.
Major May 8, 2026 [Piano Analytics] Incident supported by the on-call team 4s
Started May 08, 2026, 10:21 PM UTC· resolved May 08, 2026, 10:21 PM UTC
- investigating May 08, 2026, 10:21 PM UTC
We are currently experiencing a production incident on Piano Analytics products. Please excuse us for any inconvenience caused.
- resolved May 08, 2026, 10:21 PM UTC
The issue has been resolved and all Piano Analytics tools and interfaces are fully available. We are currently catching up the delay in real-time data retrieval.
Minor May 8, 2026 Composer C1X segmentation issues (EU & US regions) 11h 32m API Endpoints - cxense - api.cxense.com, Other
Started May 08, 2026, 06:04 AM UTC· resolved May 08, 2026, 05:36 PM UTC
- investigating May 08, 2026, 06:04 AM UTC
Composer is experiencing a service incident affecting C1X segmentation in both EU and US regions. Impact includes high error rates, degraded segment lookup, and degraded widget performance. The issue started at approximately 22:00 UTC on May 7 and continues with periods of instability. Our team is actively investigating the issue.
- monitoring May 08, 2026, 09:26 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 08, 2026, 05:36 PM UTC
The incident has been resolved.
Notice May 6, 2026 Piano Sandbox - Composer currently not working 17h 10m
Started May 06, 2026, 01:17 PM UTC· resolved May 07, 2026, 06:28 AM UTC
- identified May 06, 2026, 01:17 PM UTC
We are currently experiencing an issue with Composer in our sandbox environment; the Composer module and experiences fail to load. Our team has identified the root cause and is actively working to resolve the issue. We will provide updates as more information becomes available.
- monitoring May 06, 2026, 01:43 PM UTC
The team has deployed a fix, and the issue has been resolved. Experiences are executing, and the Composer module is working normally. We will continue monitoring the remediation.
- monitoring May 06, 2026, 01:44 PM UTC
We are continuing to monitor for any further issues.
- resolved May 07, 2026, 06:28 AM UTC
This incident has been resolved.
Minor May 5, 2026 [Activation] Increased API Error Rate on ID Endpoints (US region) 17m Registration + Login - US - id.piano.io
Started May 05, 2026, 06:48 AM UTC· resolved May 05, 2026, 07:05 AM UTC
- monitoring May 05, 2026, 08:21 AM UTC
Between 06:48 and 07:05 UTC, Piano ID API experienced elevated error rates in the US cluster. Requests requiring user authentication, token verification, and user profile retrieval were affected. Other Piano services including Composer and VX subscription management were not impacted. Service has been fully restored and all systems are operating normally. We apologize for any disruption caused during this incident.
- resolved May 05, 2026, 08:21 AM UTC
This incident has been resolved.
Major April 27, 2026 Composer waterfall issues on US environment. 16h 23m Composer Experience Execution - US
Started Apr 27, 2026, 04:57 PM UTC· resolved Apr 28, 2026, 09:20 AM UTC
- investigating Apr 27, 2026, 04:57 PM UTC
We have found an issue with our priority rules for API interaction on server/edge in Composer. The dev team is currently investigating. Clients may find their list order being changed or unsorted without any user interaction. We will continue updating this incident once we identify the root cause.
- investigating Apr 27, 2026, 06:37 PM UTC
Our team has restored the priority rules to their pre-incident state while we continue to investigate the root cause. We advise customers to avoid making changes to priority rules until the investigation is complete. We will provide further updates as more information becomes available.
- monitoring Apr 27, 2026, 10:01 PM UTC
The issue has been identified and mitigated; we will continue monitoring the remediation.
- resolved Apr 28, 2026, 09:20 AM UTC
This incident has been resolved.
- postmortem Apr 28, 2026, 02:04 PM UTC
RCA report for the incident is available at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2025-05-06-RCA-Experience-Priority-Data-Incident.pdf).
Minor April 23, 2026 Elevated server latency in Composer execution 20h 7m Composer Experience Execution - US
Started Apr 23, 2026, 02:35 PM UTC· resolved Apr 24, 2026, 10:42 AM UTC
- identified Apr 23, 2026, 02:35 PM UTC
Composer is experiencing a latency incident affecting executions in the US region starting at 10 AM UTC. Impact includes timeouts and degraded performance.
- monitoring Apr 23, 2026, 05:31 PM UTC
The incident has been resolved. We will continue monitoring the latency on the backend servers.
- monitoring Apr 23, 2026, 05:50 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 24, 2026, 10:42 AM UTC
This incident has been resolved.
Minor April 21, 2026 [Activation] Publisher Dashboard - Composer Experience version selection impacted 59m Publisher Dashboard - US - https://dashboard.piano.io, Publisher Dashboard - Asia - https://dashboard-ap.piano.io, Publisher Dashboard - Australia - https://dashboard-au.piano.io, Publisher Dashboard - Europe - https://dashboard-eu.piano.io
Started Apr 21, 2026, 09:00 AM UTC· resolved Apr 21, 2026, 09:59 AM UTC
- investigating Apr 21, 2026, 09:00 AM UTC
We are currently investigating an issue affecting the publisher dashboard where users are unable to interact with the experience version dropdown and other buttons in Composer.
- identified Apr 21, 2026, 09:22 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Apr 21, 2026, 09:59 AM UTC
This incident has been resolved. The problem was caused by a compatibility issue between two system components that were running different versions. We have updated one of the components to ensure they work together properly, which has restored normal functionality. Some users may need to clear their browser cache to fully resolve the issue and restore normal functionality. We apologize for any disruption this issue may have caused.
Minor April 19, 2026 Scheduled Maintenance: Ingress Controller Upgrade (US region) –
Started Apr 19, 2026, 11:06 PM UTC· resolved Apr 19, 2026, 11:06 PM UTC
- monitoring Apr 28, 2026, 04:00 AM UTC
THIS IS A SCHEDULED EVENT Apr 28, 04:00 - 05:00 UTC Apr 17, 15:44 UTC Scheduled - Piano is planning an infrastructure upgrade for the US region. This update is needed in order to modernize our ingress controller architecture by transitioning from ingress-nginx to F5 Nginx ingress. While we do not expect any impact, brief API disruptions (up to 15 minutes) may affect: Piano ESP API Endpoints Piano ESP Data Collection Piano ESP On-Site Recommendations Piano ESP Dashboard (US) Timings: April 28, 2026 | 04:00 - 05:00 UTC
Minor April 15, 2026 [Activation] Composer segmentation issues (EU region) 2d 1h API Endpoints - cxense - api.cxense.com, Other
Started Apr 15, 2026, 05:14 AM UTC· resolved Apr 17, 2026, 06:28 AM UTC
- investigating Apr 15, 2026, 06:09 AM UTC
Composer is experiencing a service incident affecting Audience segments and Propensity Models in the EU region. Impact includes high error rates, degraded segment lookup, and degraded widget performance.
- identified Apr 15, 2026, 06:54 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 15, 2026, 08:45 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 17, 2026, 06:28 AM UTC
This incident has been resolved.
Major April 12, 2026 ID and VX Transaction Emails not being delivered due to Internal Mandrill Issue 1d 3h Registration + Login and Management + Billing Transactional Emails
Started Apr 12, 2026, 05:00 AM UTC· resolved Apr 13, 2026, 08:46 AM UTC
- identified Apr 12, 2026, 05:00 AM UTC
Mandrill team has notified Piano about an issue impacting our account. Our team is currently coordinating efforts with the Mandrill account. The emails are queued but not getting delivered at the moment. We will continue to provide updates via this thread. Please monitor our status page for latest information.
- monitoring Apr 13, 2026, 03:32 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 13, 2026, 08:46 AM UTC
This incident has been resolved.
Minor April 9, 2026 Scheduled Maintenance: Ingress Controller Upgrade (EU region) 6d 20h
Started Apr 09, 2026, 08:16 AM UTC· resolved Apr 16, 2026, 05:00 AM UTC
- resolved Apr 16, 2026, 05:00 AM UTC
Apr 16, 05:00 UTC Completed - The scheduled maintenance has been completed. Apr 16, 04:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr 9, 08:16 UTC Scheduled - Piano is planning an infrastructure upgrade for the European region. This update is needed in order to modernize our ingress controller architecture by transitioning from ingress-nginx to F5 Nginx ingress. While we do not expect any impact, brief API disruptions (up to 15 minutes) may affect: Piano ESP API Endpoints Piano ESP Data Collection Piano ESP On-Site Recommendations Piano ESP Dashboard (EU) Timings: April 16, 2026 | 04:00 - 05:00 UTC
Minor April 9, 2026 Scheduled Maintenance: Ingress Controller Upgrade (AP region) 6d 7h
Started Apr 09, 2026, 08:14 AM UTC· resolved Apr 15, 2026, 04:00 PM UTC
- resolved Apr 15, 2026, 04:00 PM UTC
Apr 15, 16:00 UTC Completed - The scheduled maintenance has been completed. Apr 15, 15:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr 9, 08:14 UTC Scheduled - Piano is planning an infrastructure upgrade for the Asia Pacific region. This update is needed in order to modernize our ingress controller architecture by transitioning from ingress-nginx to F5 Nginx ingress. While we do not expect any impact, brief API disruptions (up to 15 minutes) may affect: Piano ESP API Endpoints Piano ESP Data Collection Piano ESP On-Site Recommendations Piano ESP Dashboard (AP) Timings: April 15, 2026 | 15:00 - 16:00 UTC
Notice April 6, 2026 [Activation] Site Licensing incident (AP region) 3h 34m Other
Started Apr 06, 2026, 01:53 PM UTC· resolved Apr 06, 2026, 05:27 PM UTC
- monitoring Apr 06, 2026, 01:53 PM UTC
Following an infrastructure update on April 3, a timezone-related bug was introduced, causing anomalies in Site Licensing, Subscription Grace Periods, and Scheduled Upgrades for clients in the Production-AP region. The root cause has been identified and a fix has been deployed. Remediation of the affected entities is in progress. A detailed Root Cause Analysis report will be shared once the incident is fully resolved, covering the scope of impact, timeline, and preventive measures.
- resolved Apr 06, 2026, 05:27 PM UTC
The incident has been resolved. A Root Cause Analysis report is currently in progress and will be shared once completed.
- postmortem Apr 28, 2026, 10:35 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/04/2026-04-06-Activation-Timezone-Configuration-Error-AP-region.pdf).
Minor March 24, 2026 [Activation] Subscription Log Performance Upgrade (US region) 6d 15h
Started Mar 24, 2026, 05:11 PM UTC· resolved Mar 31, 2026, 09:00 AM UTC
- resolved Mar 31, 2026, 09:00 AM UTC
Mar 31, 09:00 UTC Completed - The scheduled maintenance has been completed. Mar 31, 07:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar 24, 17:11 UTC Scheduled - On Tuesday, March 31, 2026, between 07:00-09:00 UTC, Piano is planning a scheduled maintenance to improve the overall performance of the Subscription Log report for the US region. During this maintenance period, the report will continue to function normally; however, there will be some data lag, with the latest changes not being immediately reflected in the report. This temporary delay is expected to last for the duration of the maintenance window, up to 2 hours. Normal, real-time data updates will resume immediately after the maintenance is completed. We appreciate your understanding and patience as we work to enhance our services.
Minor March 24, 2026 [Activation] Subscription Log Performance Upgrade (EU region) 5d 16h
Started Mar 24, 2026, 05:09 PM UTC· resolved Mar 30, 2026, 10:00 AM UTC
- resolved Mar 30, 2026, 10:00 AM UTC
Mar 30, 10:00 UTC Completed - The scheduled maintenance has been completed. Mar 30, 08:00 UTC In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar 24, 17:09 UTC Scheduled - On Monday, March 30, 2026, between 08:00-10:00 UTC, Piano is planning a scheduled maintenance to improve the overall performance of the Subscription Log report for the EU region. During this maintenance period, the report will continue to function normally; however, there will be some data lag, with the latest changes not being immediately reflected in the report. This temporary delay is expected to last for the duration of the maintenance window, up to 2 hours. Normal, real-time data updates will resume immediately after the maintenance is completed. We appreciate your understanding and patience as we work to enhance our services.
Minor March 19, 2026 Login captcha feature not working correctly 1h 30m Registration + Login - US - id.piano.io, Registration + Login - Australia - id-au.piano.io, Registration + Login - Asia - id-ap.piano.io, Registration + Login - EU - id.piano.io
Started Mar 19, 2026, 03:14 PM UTC· resolved Mar 19, 2026, 04:45 PM UTC
- investigating Mar 19, 2026, 03:14 PM UTC
We've received some complaints that end users are unable to pass the CAPTCHA verification. Our development team is currently looking into the issue.
- identified Mar 19, 2026, 03:37 PM UTC
We have identified the issue, and our team is actively working on a fix. We'll keep you updated on our progress.
- monitoring Mar 19, 2026, 03:40 PM UTC
We have successfully implemented a fix for the CAPTCHA issue. We're actively monitoring the system.
- resolved Mar 19, 2026, 04:45 PM UTC
The incident has been resolved.
Minor March 6, 2026 [Activation] Webhook Delivery Issue (US Region) 4h 55m Other
Started Mar 06, 2026, 01:10 PM UTC· resolved Mar 06, 2026, 06:05 PM UTC
- identified Mar 06, 2026, 01:10 PM UTC
We’re currently experiencing an issue affecting webhook deliveries in the US region. Some webhooks may fail or contain invalid data. Our team has identified the cause and is actively working on a fix. Updates will follow as more information becomes available.
- identified Mar 06, 2026, 01:27 PM UTC
We have identified an issue causing corrupted webhook payloads for a subset of users. Due to a database limitation where a foreign key column hit its maximum value, webhooks may contain corrupted metadata fields, including malformed data like '*mac.com' instead of correct information. Our team is actively working on a fix, and we'll keep you updated on our progress.
- monitoring Mar 06, 2026, 01:43 PM UTC
We have successfully implemented a fix for the webhook payload issue. New events are now being delivered with correct content again. To expedite the resolution, events were rerouted to a new table, which means historical events from before 13:31 UTC are temporarily unavailable while we work on restoring them. We're actively monitoring the system and continuing restoration efforts, though our preliminary assessment indicates that data from the affected time period (11:20 UTC - 13:31 UTC) may not be completely recoverable.
- resolved Mar 06, 2026, 06:05 PM UTC
The issue has been resolved, and the team will continue working on restoring the missing events.
- postmortem Mar 30, 2026, 08:48 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/03/2026-03-06-RCA-Activation-Webhooks-incident.pdf).
Notice February 19, 2026 Amplifier performance issues – SocialFlow.com Site uptime, SocialFlow publishing
Started Feb 19, 2026, 10:00 PM UTC· resolved Feb 19, 2026, 10:00 PM UTC
- resolved Feb 20, 2026, 10:12 AM UTC
On February 19, 2026 between 12pm and 10pm UTC, Piano experienced a production incident affecting Amplifier (Socialflow). The incident impacted compose box functionality for a subset of users (browser-dependent). Some users also experienced degraded UI performance. Additionally, publishing was affected for users who rely on the compose box for manual post publishing. The issue has been resolved by rolling back Socialflow Web to the previous release. We are currently investigating the root cause and will publish an incident report once the analysis is complete.
- postmortem Mar 12, 2026, 12:40 PM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/03/2026-02-19-RCA-Amplifier-degraded-performance.pdf).
Minor January 19, 2026 Piano Crawler not crawling 8h 29m Piano Data Collection - Crawler
Started Jan 19, 2026, 12:05 AM UTC· resolved Jan 19, 2026, 08:35 AM UTC
- investigating Jan 19, 2026, 12:05 AM UTC
Piano Crawler has not been crawling for the last 24 hours. We are currently investigating the issue.
- investigating Jan 19, 2026, 02:23 AM UTC
We are continuing to investigate this issue.
- investigating Jan 19, 2026, 03:55 AM UTC
We are still looking into this issue, but the investigation shows that this affects only a small number of sites, not all of them.
- identified Jan 19, 2026, 06:54 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jan 19, 2026, 07:22 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 19, 2026, 08:35 AM UTC
This incident has been resolved.
- postmortem Jan 28, 2026, 08:50 AM UTC
RCA report for the incident can be found at [this link](https://docs.piano.io/wp-content/uploads/2026/01/2026-01-19-RCA-Piano-Crawler-not-crawling.pdf)
Minor January 12, 2026 Composer segmentation issues [EU region] 13h 4m Other
Started Jan 12, 2026, 12:07 PM UTC· resolved Jan 13, 2026, 01:11 AM UTC
- investigating Jan 12, 2026, 12:07 PM UTC
Composer is experiencing a service incident affecting DMP segments and Propensity Models in the EU region. Impact includes high error rates, degraded segment lookup, and degraded widget performance.
- identified Jan 12, 2026, 02:19 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jan 12, 2026, 02:55 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 13, 2026, 01:11 AM UTC
This incident has been resolved.
Critical December 24, 2025 Elevated API Errors 7h 22m Publisher Dashboard - US - https://dashboard.piano.io, Checkout - US - buy.piano.io, API Endpoints - US - api.piano.io, Composer Experience Execution - US
Started Dec 24, 2025, 02:21 PM UTC· resolved Dec 24, 2025, 09:44 PM UTC
- investigating Dec 24, 2025, 02:21 PM UTC
Piano platform is experiencing an elevated level of API errors. Our engineers are currently looking into the issue.
- identified Dec 24, 2025, 02:43 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Dec 24, 2025, 02:53 PM UTC
A fix has been implemented, and we're monitoring its results
- resolved Dec 24, 2025, 09:44 PM UTC
The incident has been resolved.
- postmortem Jan 07, 2026, 08:43 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2026/01/2025-12-24-RCA-Activation-outage.pdf).
Major December 16, 2025 [Piano Analytics] Data Ingestion Issue 17h 59m Piano Analytics Data Processing
Started Dec 16, 2025, 03:30 AM UTC· resolved Dec 16, 2025, 09:30 PM UTC
- investigating Dec 16, 2025, 06:35 AM UTC
We are currently experiencing a production incident on Piano Analytics products. A major incident is currently ongoing with our cloud provider in the EU regions, impacting both streaming data ingestion and data consolidation processes. Our data collection and processing are not impacted. The issue started at approximately 03:30 UTC on December 16, 2025, and is still ongoing at this time on their side. Please excuse us for any inconvenience caused.
- investigating Dec 16, 2025, 08:54 AM UTC
We are continuing to investigate this issue.
- identified Dec 16, 2025, 10:40 AM UTC
Our provider's service has been partially restored. Our teams are now actively working on completing the recovery on our side, including relaunching data loads and catching up on delayed processing.
- resolved Dec 17, 2025, 09:23 AM UTC
The issue has been resolved and all (Piano Analytics) tools and interfaces are fully available. Some data still needs to be sent for some specific customers, but we will communicate with them directly.
- postmortem Dec 17, 2025, 09:23 AM UTC
**Incident explanation:** A major incident has occurred with our cloud provider in the EU regions, impacting both streaming data ingestion and data consolidation processes. Our data collection and processing were not impacted. **Actions performed:** Once the provider service was restored, we checked the status of our system and restarted whatever needed to be restarted. Everything gradually returned to normal. **Actions to come:** We still need to address a few data transfer issues for specific customers on a case-by-case basis (particularly dataflow exports).Everything should be back to normal by this morning.
Critical December 5, 2025 Piano service disruption due to Cloudflare outage 42m Publisher Dashboard - US - https://dashboard.piano.io, API Endpoints - US - api.piano.io, Publisher Dashboard - Asia - https://dashboard-ap.piano.io, API Endpoints - Asia - api-ap.piano.io and 4 more
Started Dec 05, 2025, 08:47 AM UTC· resolved Dec 05, 2025, 09:30 AM UTC
- identified Dec 05, 2025, 09:02 AM UTC
We are currently experiencing a service disruption affecting our services due to an outage on Cloudflare's side. As a result, clients may also experience difficulties logging into Piano dashboards. Our team is actively monitoring the situation. We apologize for any inconvenience this may cause and appreciate your patience. Further updates will be provided as soon as more information becomes available. For reference: https://www.cloudflarestatus.com/
- identified Dec 05, 2025, 09:10 AM UTC
We are currently experiencing a service disruption affecting our services due to an outage on Cloudflare's side. For reference: https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q
- monitoring Dec 05, 2025, 09:17 AM UTC
Cloudflare has fixed the issue on their end (https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q), and we are now observing incoming requests on our systems again. We will continue to monitor the situation and provide updates if anything changes.
- resolved Dec 05, 2025, 09:30 AM UTC
This incident has been resolved.
Minor December 3, 2025 CXense - Delays in ingestion of user profile updates 19h 31m API Endpoints - cxense - api.cxense.com
Started Dec 03, 2025, 08:15 PM UTC· resolved Dec 04, 2025, 03:46 PM UTC
- identified Dec 03, 2025, 08:15 PM UTC
We have identified an issue affecting the ingestion of user profile updates, causing processing delays. Our engineering team has implemented remediation measures, and we are currently working to process the backlog of delayed updates.
- identified Dec 03, 2025, 09:14 PM UTC
Backlog of delayed updates have been processed. We continue to monitor the situation.
- monitoring Dec 03, 2025, 09:39 PM UTC
We continue to monitor the situation.
- resolved Dec 04, 2025, 03:46 PM UTC
This incident has been resolved.
- postmortem Dec 17, 2025, 08:52 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2025/12/2025-12-03-RCA-Delays-in-ingestion-of-user-profile-updates.pdf).