FullStory experienced a notice incident on October 31, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 31, 2024, 09:33 PM UTC
From October 31, 2024 at 2:38PM ET through October 31, 2024 at 4:07PM ET, the Fullstory APIs were unintentionally unavailable. The majority of requests to Fullstory APIs would have returned an error. Fullstory customers that attempted to send an API request during the outage window will need to process these requests again. If you believe your account was impacted or have any questions, please reach out to [email protected].
- postmortem Nov 07, 2024, 09:07 PM UTC
On October 31, 2024, our API service experienced a disruption that resulted in a significant number of dropped requests and errors. The incident lasted for approximately 1.5 hours and impacted all users of our public server API. # Customer Impact Most public server API requests sent during the incident time frame \(18:37:29 UTC to 20:12:39 UTC\) were unsuccessful and should be re-submitted. We acknowledge the inconvenience and disruption this caused our customers and sincerely apologize for any impact on your operations. # Root Cause The root cause of the incident was a decrease in the capacity of our API service due to an unsuccessful deployment. This decrease in capacity, combined with higher than usual request volume, led to the service being unable to handle the incoming traffic. # Resolution Upon discovery of the issue, our on-call operations team was immediately notified and responded swiftly. The team's actions included re-executing the failed deployment to restore normal API service functionality. # Process Changes and Prevention **Actions Taken:** * We re-executed the unsuccessful deployment. * We permanently scaled up the capacity of our API service to handle the increased traffic. * We are implementing additional monitoring and alerting to detect and prevent similar issues in the future. **Ongoing Improvements:** * We are reviewing our deployment processes to minimize the risk of similar incidents. * We are investigating ways to improve the resilience of our API service to handle unexpected spikes in traffic. We deeply regret this incident and invite any Fullstory customer who was materially affected to contact [[email protected]](mailto:[email protected]). We stand by ready to fully address all of your concerns.