Piano.io incident
Piano Audience/Insight: UI, /traffic API unavailable
Piano.io experienced a major incident on June 23, 2025 affecting API Endpoints - cxense - api.cxense.com and Piano Insight Dashboard and 1 more component, lasting 1d 7h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 23, 2025, 09:27 AM UTC
/traffic/* APIs are currently unavailable. Due to this incident, Piano Insight UI is unavailable, and Audience UI is also partially affected.
- identified Jun 23, 2025, 09:56 AM UTC
The issue has been identified and a fix is in progress.
- identified Jun 23, 2025, 12:46 PM UTC
Our team is continuing to investigate the issue and expects to provide a new update within the next two hours.
- monitoring Jun 23, 2025, 01:51 PM UTC
An initial fix has been deployed that has reduced error rates to zero. Our team is applying additional changes to ensure stability and monitoring the results.
- identified Jun 23, 2025, 02:31 PM UTC
Our team is still seeing issues after the initial fix and is continuing to pursue other solutions. We will continue to provide updates via this channel.
- identified Jun 24, 2025, 05:49 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Jun 24, 2025, 08:08 AM UTC
A fix has been applied, and we are observing a gradual restoration of functionality to normal levels. We are actively monitoring the results.
- resolved Jun 24, 2025, 05:18 PM UTC
We observed full restoration of the dashboard and API availability for Insights. This incident has been resolved.
- postmortem Jul 07, 2025, 01:28 PM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2025/07/2025-06-23-RCA-Piano-Audience_Insight-UI-traffic-API-unavailable.pdf).