Cowbell incident
US Production instance degradation
Cowbell experienced a minor incident on January 30, 2025, lasting 4d 3h. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jan 30, 2025, 06:28 PM UTC
Status: Investigating Ongoing Production Performance Issue** We are currently investigating a performance degradation affecting our US Production instance. Our engineering team is actively working to identify and resolve the issue. At this time, users may experience intermittent access or performance challenges when using our console. We understand the critical nature of this disruption and are treating this with our highest priority. We will provide updates as we learn more about the root cause and estimated time to resolution. **Recommended Action:** Users may want to temporarily retry accessing the console or wait for further updates. **Last Updated:** 2025-01-30 18:20:36 UTC
- resolved Jan 30, 2025, 06:47 PM UTC
Status Update: US Production Instance Performance Resolved** **Update Details:** - Issue: Performance degradation in US Production instance - Root Cause: Networking conflict - Current Status: Mitigated - Duration: Approximately 29 minutes (18:17 - 18:46 UTC) **Customer Impact:** During the incident, users may have experienced intermittent access and performance challenges with the console. The issue has now been resolved, and the system should be functioning normally. **Recommended Action:** No further action is required. The console is now accessible and performing as expected. **Last Updated:** 2025-01-30 18:46 UTC Key changes from the incident: - Initial detection of performance issues - Active investigation by engineering team - Identification of networking conflict - Successful mitigation of the performance problem - Full restoration of service functionality