Piano.io Outage History
Piano.io is up right nowPingoru has recorded 84 incidents for Piano.io: 80 in the last 12 months and 13 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 15 | 231 of 720 (32%) |
| May 2026 | 10 | 81 of 744 (11%) |
| June 2026 | 2 | 34 of 720 (5%) |
| July 2026 | 5 | 173 of 744 (23%) |
| August 2026 | 5 | 22 of 744 (3%) |
| September 2026 | 5 | 18 of 720 (2%) |
| October 2026 | 3 | 23 of 591 (4%) |
Incidents, newest first
Minor December 2, 2025 [Piano Content] Crawling delay 10h 58m Piano Data Collection - Crawler
Started Dec 02, 2025, 09:02 AM UTC· resolved Dec 02, 2025, 08:01 PM UTC
- identified Dec 02, 2025, 09:02 AM UTC
We are currently experiencing an incident where newly published articles are not crawled immediately due to an accumulated event lag. Our engineers identified the issue and promptly applied a fix. The system is clearing the processing backlog, and normal crawling should resume within a few hours.
- monitoring Dec 02, 2025, 01:02 PM UTC
Crawler processing is now functioning normally. However, we are still observing an approximate 4-hour delay. We are closely monitoring the situation and will update once the lag has been fully resolved.
- resolved Dec 02, 2025, 08:01 PM UTC
The issue has been resolved, and crawling services are operating normally.
- postmortem Dec 09, 2025, 07:27 AM UTC
RCA report for the incident can be found at [**this link**](https://docs.piano.io/wp-content/uploads/2025/12/2025-12-02-RCA-Piano-Content_-Crawling-delay.pdf).
Critical November 28, 2025 Piano Insight is experience a production incident 2h 50m Piano Audience - Insight
Started Nov 28, 2025, 01:54 AM UTC· resolved Nov 28, 2025, 04:45 AM UTC
- investigating Nov 28, 2025, 03:26 AM UTC
We noticed a significant traffic drop in Insight started on November 28, 2025 01:54 UTC. Our engineers are investigating the issue.
- monitoring Nov 28, 2025, 03:54 AM UTC
Our engineers have identified the root cause and have implemented the fix. We are monitoring the results.
- resolved Nov 28, 2025, 04:45 AM UTC
This incident has been resolved.
Minor November 22, 2025 Traffic drop - Piano Insight 2d 3h API Endpoints - cxense - api.cxense.com, Piano Audience - Insight
Started Nov 22, 2025, 05:12 AM UTC· resolved Nov 24, 2025, 08:25 AM UTC
- identified Nov 22, 2025, 05:12 AM UTC
We noticed a significant traffic drop in Insight started on November 22, 2025 02:37 UTC. The technical team identified the root cause and is now working on the resolution.
- monitoring Nov 22, 2025, 05:47 AM UTC
Monitoring - The issue has been resolved, and services are operating normally. We are continuing to monitor the situation to ensure ongoing stability.
- resolved Nov 24, 2025, 08:25 AM UTC
This incident has been resolved.
Notice October 29, 2025 Elevated 400 errors for Server Composer Experiences (US & EU regions) – Composer Experience Execution - US, Composer Experience Execution - Europe
Started Oct 29, 2025, 07:27 AM UTC· resolved Oct 29, 2025, 07:27 AM UTC
- resolved Oct 29, 2025, 07:27 AM UTC
Piano detected elevated 400 errors on API Interaction for Server Composer Experiences between 6:22 AM - 6:43 AM UTC in the US and EU regions. This was caused by a new release that introduced a serialization logic change, resulting in unexpected object graph handling. The issue was auto-detected and immediately resolved via our monitoring system. We apologize for any inconvenience caused and will enhance automated smoke testing for server-side experiences to prevent similar issues.
Critical October 24, 2025 [Piano Analytics] Potential interface availability issues 22m Piano Analytics - Reporting API, Piano Analytics Interfaces Availability
Started Oct 24, 2025, 08:22 AM UTC· resolved Oct 24, 2025, 08:44 AM UTC
- investigating Oct 24, 2025, 08:22 AM UTC
We are currently experiencing availability issues with our API tools and interfaces. Our technical teams are actively working to resolve the issue. We apologize for any inconvenience caused. Affected customers - All customers
- resolved Oct 24, 2025, 08:44 AM UTC
The issue has been resolved and all our tools and interfaces are fully available. Data collection was not affected, all events were properly ingested.
- postmortem Oct 24, 2025, 08:44 AM UTC
**Incident explanation:** A production release caused temporary inaccessibility of our interfaces. **Actions performed:** A rollback was performed to restore service. **Actions to come:** No more actions to come.
Major October 20, 2025 Some of Piano services in US environment affected by AWS incident 5h 42m Checkout - US - buy.piano.io, API Endpoints - US - api.piano.io, Composer Experience Execution - US, Registration + Login - US - id.piano.io and 2 more
Started Oct 20, 2025, 08:09 AM UTC· resolved Oct 20, 2025, 01:52 PM UTC
- investigating Oct 20, 2025, 08:09 AM UTC
We are currently investigating this issue.
- investigating Oct 20, 2025, 08:38 AM UTC
We are continuing to investigate this issue.
- investigating Oct 20, 2025, 08:41 AM UTC
We are continuing to investigate this issue.
- investigating Oct 20, 2025, 09:53 AM UTC
We are continuing to investigate this issue.
- monitoring Oct 20, 2025, 09:56 AM UTC
A fix has been implemented and we are monitoring the results.
- investigating Oct 20, 2025, 10:40 AM UTC
We are currently investigating this issue.
- monitoring Oct 20, 2025, 11:17 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 20, 2025, 01:52 PM UTC
This incident has been resolved.
- postmortem Oct 23, 2025, 10:31 AM UTC
# AWS US-EAST-1 Outage — Public RCA **Date:** 2025-10-20 **Status:** Resolved ## Summary On 2025-10-20, an outage in the AWS US-EAST-1 region caused interruptions to customer purchases, email delivery, reporting, and our sandbox environment. Core customer-facing services were restored by 09:20 UTC; sandbox services were fully restored by 17:30 UTC. We determined the outage was caused by multiple simultaneous infrastructure failures in the region that prevented our systems from processing requests and scaling to meet load. ## Impact and Timeline \(UTC\) * **06:50 - 09:20** Purchase transactions failed * **07:25 - 09:20** Email delivery was delayed * **07:35 - 09:25** Composer experienced degraded performance, particularly in server-side experiences and metering, which may have caused delays in applying updates * **06:50 - 11:05** Reporting that depends on our US-based API endpoints was unavailable across regions * **12:30 - 17:30** Sandbox environment unavailable due to lack of capacity ## Root Cause A combination of AWS service disruptions in US-EAST-1 prevented our systems from operating and scaling normally: * **DynamoDB service errors** blocked payment processing that relies on DynamoDB for scalable transaction state * **SQS queue failures** prevented queue processing used for email delivery and reporting workflows * **EC2 API disruptions** and widespread Spot instance reclamation prevented us from provisioning new instances. Existing capacity was constrained; in several cases it was unsafe to force-restart instances * **S3 failures** prevented timely application of configuration updates required to restore some services ## Actions Taken During the Incident * Immediately mobilized all OPS and on-call engineers to triage and mitigate * Engaged directly with our AWS account team and exchanged frequent status updates with AWS support * Tuned Karpenter to expand instance allocation across additional availability zones within the region to restore capacity where possible * Monitored system behavior continuously and evaluated the need for a cross-region disaster recovery \(DR\) activation; determined DR would carry risk of partial data loss and was not necessary once regional capacity was restored ## Resolution After AWS services gradually came back online, we restored capacity and brought services back online. Core customer-facing services \(purchases, emails, reporting\) were restored by 09:20 UTC; the sandbox environment returned to normal by 17:30 UTC after additional capacity and configuration propagation completed. ## Next Steps and Mitigations We are implementing the following actions to reduce the risk and impact of similar incidents: * Conduct a joint post-incident review with AWS to identify specific failure modes and remediation steps on their side * Add additional resilience for DynamoDB-backed flows \(e.g., improved fallback paths\) * Reduce reliance on Spot capacity for sandbox services * Improve S3 configuration propagation and rollback capabilities to avoid blocking recoveries * Consider simplifying critical email delivery in case queues are not available \(usual cross-regional queue rerouting would not work in this case\) ## Q&A **Q: Why didn't we have any failover mechanisms or backup plans in case anything like this happened?** A: Mitigations \(rerouting services\) were blocked by AWS-based dynamic reconfiguration subsystem failures. Full DR was considered overkill with too serious potential consequences \(extended full downtime, potential data loss\). **Q: Do we plan on improving for the future to better handle situations like these in terms of triggers or warnings beforehand?** A: We had monitors alerting almost immediately, so on-call engineers started to look into the issue right away. For plans, see the mitigations section above. **Q: Why did it take so long for Sandbox to be live again?** A: AWS did not have on-demand server instances available, and Spot instances were withdrawn. We couldn't reuse production cluster nodes for sandbox because the production cluster itself had considerable resource over-utilization due to the lack of available servers. --- We recognize the impact this incident had on customers and are committed to improving the resilience and availability of our platform. We will follow up with the findings from the joint post-incident review with AWS and a timeline for the planned mitigations.
Notice October 16, 2025 Sandbox unavailable 4h 48m Other
Started Oct 16, 2025, 02:21 AM UTC· resolved Oct 16, 2025, 07:09 AM UTC
- investigating Oct 16, 2025, 02:21 AM UTC
Sandbox.piano.io is currently unavailable. We apologize for any inconvenience caused.
- monitoring Oct 16, 2025, 07:01 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 16, 2025, 07:09 AM UTC
This incident has been resolved.