Piano.io experienced a major incident on November 14, 2025 affecting API Endpoints - cxense - api.cxense.com and Insight Data collection and 1 more component, lasting 3d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 14, 2025, 08:19 AM UTC
We noticed a delay in displaying traffic data in Insight today morning again. Our team started investigating the issue.
- identified Nov 14, 2025, 08:23 AM UTC
An issue found with data cubes in one of our data centers. We are working on the resolution now.
- monitoring Nov 14, 2025, 11:48 AM UTC
An issue was resolved and traffic data delay is now mitigated. We are now monitoring the performance of the data cubes.
- investigating Nov 14, 2025, 04:12 PM UTC
While monitoring the remediation, we started to see an increase in errors in segmentation for c1x segments and delays in our traffic data. Our team is currently investigating the issue.
- investigating Nov 14, 2025, 05:14 PM UTC
We are experiencing an issue with data cubes again, our team is now investigating.
- monitoring Nov 15, 2025, 05:40 AM UTC
The issue has been resolved, and services are operating normally. We are continuing to monitor the situation to ensure ongoing stability.
- resolved Nov 18, 2025, 08:02 AM UTC
The incident has been resolved. We are now conducting a thorough root cause analysis, and we will share a detailed report once it becomes available.
- postmortem Dec 02, 2025, 08:15 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2025/12/2025-11-13-RCA-Traffic-drop-Piano-Insight.pdf).