TrueContext experienced a minor incident on October 2, 2025 affecting Admin Console and Form Builder and 1 more component, lasting 4h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 02, 2025, 02:37 PM UTC
We are currently investigating this issue.
- identified Oct 02, 2025, 02:37 PM UTC
The issue has been identified and a fix is being implemented.
- identified Oct 02, 2025, 02:38 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Oct 02, 2025, 05:26 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 02, 2025, 06:20 PM UTC
This incident has been resolved.
- postmortem Oct 02, 2025, 07:26 PM UTC
**Service Degradation Update - October 1-2, 2025** Given the service degradation over the past two days, we wanted to provide detail on what happened and our path forward. **Incident 1 - October 1** At 11:30am EDT, we detected excessive pressure on our persistence store. We immediately rolled back a change released earlier that morning and performed manual actions to reduce system pressure. Service incrementally recovered to normal operation within one hour. We removed all changes from the prior 24 hours to investigate further, and those changes currently remain removed. **Incident 2 - October 2** At 10:00am EDT, approximately 30 minutes after deploying a new changeset, we noticed degraded response times, though not as severe as the prior day. We immediately initiated a rollback, which did not resolve the issue. A second, more extensive rollback also had no effect. Further analysis indicated the issue was likely triggered by increased activity volume during weekday morning hours rather than our code deployments. Given the reduced severity of this incident, we were able to investigate more thoroughly while maintaining acceptable service levels. We manually restored full system operation by early afternoon. **Current Status and Next Steps** * We have identified potential root causes and are actively investigating * Out of an abundance of caution, all code deployments are paused until we conclusively identify the root cause * We are implementing additional monitoring to detect similar issues earlier * We have created an additional response playbook with mitigation measures to resolve this type of incident more quickly We apologize for the disruption this has caused. We are committed to identifying and safely resolving the underlying root causes to prevent recurrence.