StoryStream incident
Disruption affecting platform and embedded modules
Resolved on October 22, 2025.
What Happened This morning, our platform experienced service disruptions that affected both our main platform and embedded modules deployed on customer sites. Users may have experienced: - Slow loading times or timeouts - Intermittent errors when accessing content - Degraded performance of embedded widgets and modules Root Cause Our investigation revealed that a surge of specific request types overwhelmed our cache layer, which exposed an underlying performance issue in our content pipeline. A slow-running database query, which normally would have been handled efficiently by our caching system, became a bottleneck when the cache was unable to serve requests effectively. Resolution and Immediate Actions - Immediate mitigation: We quickly identified and resolved the cache overload issue, restoring normal service by 11:00 AM - Query optimization: The problematic slow query has been optimized to prevent future performance impacts Preventive Measures We are implementing additional safeguards to prevent similar incidents: 1. Enhanced cache resilience: Improving our cache layer's ability to handle traffic spikes and specific request patterns 2. Cache header improvements: We've discovered that our cache headers weren't being applied consistently in all scenarios. We're fixing these gaps to provide better protection against similar issues 3. Monitoring enhancements: Adding more granular monitoring to detect and alert on cache performance issues before they impact customers Our Commitment We sincerely apologize for any inconvenience this incident may have caused. We take the reliability of our platform seriously and are committed to continuous improvement of our infrastructure and monitoring systems. If you experienced any specific issues during this time or have questions about this incident, please don't hesitate to reach out to our support team.
Update timeline
- resolved Oct 22, 2025, 01:31 PM UTC
What Happened This morning, our platform experienced service disruptions that affected both our main platform and embedded modules deployed on customer sites. Users may have experienced: - Slow loading times or timeouts - Intermittent errors when accessing content - Degraded performance of embedded widgets and modules Root Cause Our investigation revealed that a surge of specific request types overwhelmed our cache layer, which exposed an underlying performance issue in our content pipeline. A slow-running database query, which normally would have been handled efficiently by our caching system, became a bottleneck when the cache was unable to serve requests effectively. Resolution and Immediate Actions - Immediate mitigation: We quickly identified and resolved the cache overload issue, restoring normal service by 11:00 AM - Query optimization: The problematic slow query has been optimized to prevent future performance impacts Preventive Measures We are implementing additional safeguards to prevent similar incidents: 1. Enhanced cache resilience: Improving our cache layer's ability to handle traffic spikes and specific request patterns 2. Cache header improvements: We've discovered that our cache headers weren't being applied consistently in all scenarios. We're fixing these gaps to provide better protection against similar issues 3. Monitoring enhancements: Adding more granular monitoring to detect and alert on cache performance issues before they impact customers Our Commitment We sincerely apologize for any inconvenience this incident may have caused. We take the reliability of our platform seriously and are committed to continuous improvement of our infrastructure and monitoring systems. If you experienced any specific issues during this time or have questions about this incident, please don't hesitate to reach out to our support team.
Your product runs on more than StoryStream
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services