Products Up experienced a major incident on December 12, 2024 affecting Data Processing, lasting 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 12, 2024, 09:14 AM UTC
We are currently experiencing a partial outage of the Download Proxy service. Users have reported issues with accessing the service, with some connections being rejected. This appears to be related to recent changes in IP addressing for our proxy services. We've implemented a fix and the download proxy should function again as intended.
- resolved Dec 12, 2024, 09:19 AM UTC
The partial outage affecting the Download Proxy service has been successfully resolved. The issue was traced back to misconfigurations in the firewall settings following recent changes to IP addressing for our proxy services. As the service is going live again, we are committed to enhancing our systems to prevent similar incidents in the future. Thank you for your understanding and support.
- postmortem Dec 12, 2024, 09:20 AM UTC
**Incident Overview** The Download Proxy experienced a partial outage lasting a few hours due to issues with our firewall configuration. This incident resulted in the rejection of traffic directed to the Download Proxy, impacting user access and service availability. **Root Cause** The root cause of the outage was a misconfiguration in the firewall settings following recent changes in IP addressing for our proxy services. The firewall began rejecting legitimate traffic, leading to service unavailability. Since the outage was partial, only some connections were affected, making immediate detection difficult. **Impact** Users were unable to access the Download Proxy for several hours, but the impact was not uniform across all connections. Some users experienced delays in their workflows, while others were unaffected. **Preventive Measures** To prevent similar incidents in the future, the following measures will be implemented: Change Management Process: Establish a more robust change management process to review potential impacts of IP changes on firewall and proxy configurations. Automated Alerts: Set up automated alerts for unusual traffic rejections to allow for quicker response times. Documentation: Ensure thorough documentation of any changes made to network configurations, including IP address assignments. **Conclusion** The partial outage of the Download Proxy was a result of firewall misconfiguration following IP address changes. By implementing the aforementioned preventive measures, we aim to enhance our resilience to similar issues in the future and ensure better service reliability for our users.