Service Degradation on January 30, 2026

Update timeline

1. resolved Feb 04, 2026, 02:07 PM UTC

   ### What Happened On January 30, we deployed a network configuration change designed to block aggressive web crawlers from China that were placing unnecessary load on our infrastructure. Unfortunately, this change contained a bug that caused some requests to our service to be handled incorrectly. During the affected period, some customers may have experienced: - **Status notice updates not being saved** when submitted - **General performance degradation** across the platform ### Timeline - **12:48 PM UTC:** Configuration deployed to production - **3:30 PM UTC:** Issue discovered by our engineering team - **4:32 PM UTC:** Configuration rolled back, normal service resumed ### Why This Happened The bug slipped through our standard quality assurance process because it affected network infrastructure that wasn't covered by our existing automated testing. While our performance monitoring detected the degradation, the threshold wasn't set to trigger an immediate alert. ### What We've Done We've taken the following steps to prevent similar issues: 1. **Improved monitoring thresholds** to alert our team immediately when performance degrades 2. **Expanded automated testing** to include network-level infrastructure changes that were previously outside our QA scope 3. **Conducted a full review** of the change to understand what went wrong ### Impact Assessment We're grateful that no customers reported issues during this incident, suggesting the actual impact was minimal. Importantly, **no data was lost** - any failed form submissions would have been immediately apparent to users, who could simply retry. ### Moving Forward We've fixed the issue and improved our safeguards to prevent this from happening again. If you have any questions about this incident, feel free to reach out.