Resolved after 1 day, on June 9, 2026.
**Incident explanation:** The latency was caused by an issue on our cloud data platform provider's side. An internal infrastructure configuration change introduced resource contention that affected workloads with high ingestion concurrency, leading to elevated response times and intermittent timeouts when loading files. The change has been reverted by the provider, and ingestion performance has fully returned to its normal baseline. **Actions performed:** Opened and tracked an incident with our data platform provider and provided detailed monitoring evidence to support their investigation.Deployed mitigations on our streaming ingestion stack (increased service scaling and improved handling of stalled ingestion calls) to limit customer impact during the incident.Temporarily tuned ingestion timeout and retry settings to keep file loading as stable as possible while the root cause was being addressed.Continuously monitored ingestion latency and error rates until performance returned to normal.Confirmed full recovery, then reverted the temporary configuration to restore standard behaviour and avoid masking any future issues. **Actions to come:** Continued close monitoring of ingestion latency over the coming days to confirm lasting stability.Keep the temporary mitigation patterns configurable in the code so they can be re-applied quickly should a similar provider-side degradation occur.
Update timeline
- investigating Jun 09, 2026, 09:52 AM UTC
Since approximately 11:00 PM (UTC) on June 8, 2026, there have been intermittent delays loading files into Snowflake. The average load duration is currently about 10 minutes, compared to the usual under 2 minutes. The issue has been identified and a ticket has been opened with Snowflake.
- resolved Jun 10, 2026, 09:55 AM UTC
The incident has been resolved and is no longer affecting Piano Analytics. We apologise for the inconvenience caused.
- postmortem Jun 10, 2026, 09:55 AM UTC
**Incident explanation:** The latency was caused by an issue on our cloud data platform provider's side. An internal infrastructure configuration change introduced resource contention that affected workloads with high ingestion concurrency, leading to elevated response times and intermittent timeouts when loading files. The change has been reverted by the provider, and ingestion performance has fully returned to its normal baseline. **Actions performed:** Opened and tracked an incident with our data platform provider and provided detailed monitoring evidence to support their investigation.Deployed mitigations on our streaming ingestion stack (increased service scaling and improved handling of stalled ingestion calls) to limit customer impact during the incident.Temporarily tuned ingestion timeout and retry settings to keep file loading as stable as possible while the root cause was being addressed.Continuously monitored ingestion latency and error rates until performance returned to normal.Confirmed full recovery, then reverted the temporary configuration to restore standard behaviour and avoid masking any future issues. **Actions to come:** Continued close monitoring of ingestion latency over the coming days to confirm lasting stability.Keep the temporary mitigation patterns configurable in the code so they can be re-applied quickly should a similar provider-side degradation occur.
Your product runs on more than Piano.io
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services