Products Up Outage History

Started Oct 16, 2026, 11:05 AM UTC· resolved Oct 21, 2026, 01:28 PM UTC

1. investigating Oct 16, 2026, 11:05 AM UTC

   Since 13 October, we have seen elevated load on our data processing infrastructure. Some customers may see longer run times, and runs may be delayed, especially during peak hours. Imports, exports and runs still complete, but they take longer than usual. Our engineering team is actively investigating and working to restore normal processing times. No data is lost. We will post another update by 20:00 CEST, or sooner if anything changes. We apologize for the inconvenience.

2. investigating Oct 16, 2026, 06:09 PM UTC

   We are still seeing elevated load on our data processing infrastructure. Imports, exports and runs continue to complete, but run times are longer than usual, especially during peak hours. Our load has stayed at this level since 13 October. The increase is linked to a significant growth in the volume of data being processed on the platform. Because the load has not eased, customers should expect delays to continue through the weekend. Runs that depend on strict timing, such as scheduled exports with a delivery deadline, may be affected. We will post our next update if the situation changes. In the meantime, we apologize again for the inconvenience and thank you for your continued support.

3. resolved Oct 21, 2026, 01:28 PM UTC

   Processing times have been back to normal since Monday, 19 October, at about 17:00 CEST. We have monitored the platform since then and confirmed that imports, exports and runs are completing at their usual speed, including during peak hours. Between 13 and 19 October, a significant increase in the volume of data being processed on our platform used up our processing capacity. As a result, runs took longer than usual and some were delayed, especially at peak times. Once the source of the additional volume was identified, it was paused, and processing capacity returned to normal. To detect this kind of capacity pressure earlier, we have added new monitoring of processing load and queue build-up. We apologize for the inconvenience. If you still see unexpected delays on your sites, please contact our Support team.

Read the full incident report →

Started Aug 05, 2026, 01:21 PM UTC· resolved Aug 05, 2026, 04:59 PM UTC

1. investigating Aug 05, 2026, 01:21 PM UTC

   We're observing queues in our data processing scheduler, all slots are currently full. As a result, site run start may be delayed from schedule by a few minutes. We are currently investigating the issue and will keep you informed from further developments. Best regards Your Productsup Technical Operations Team

2. identified Aug 05, 2026, 03:12 PM UTC

   Dear customers, we've tracked the issue down to a faulty CPU on one of our network equipments. We have shut it down for now, the traffic will flow gracefully through another path. The queue should drain down in the next hour. We appreciate your patience through this incident and thank you for your continued support. Your Productsup Technical Operations Team

3. monitoring Aug 05, 2026, 04:21 PM UTC

   Dear customers, The scheduler queue has gone back to zero since 17:30 CEST. We are currently working with our provider to fix the defective network equipment in order for us to go back to full capacity. When this operation is complete, we will close this incident. Best regards Your Productsup Technical Operations Team

4. resolved Aug 05, 2026, 04:59 PM UTC

   Dear customers, Our faulty network equipment has been repaired and we are back at full capacity, therefore we will be closing this issue. Best regards Your Productsup Technical Operations Team

Read the full incident report →

Started Jul 07, 2026, 08:24 AM UTC· resolved Jul 07, 2026, 09:26 AM UTC

1. investigating Jul 07, 2026, 08:24 AM UTC

   We are currently investigating an issue causing "unauthorized" error responses when users attempt to access the Platform user interface. Current Impact: Users may be unable to log in or view the UI dashboard. Data Processing: Fully functional and not affected. Our engineering team is actively working to resolve this. We apologize for the disruption and will provide an update as soon as we have more information.

2. resolved Jul 07, 2026, 09:26 AM UTC

   The issue causing unauthorized error responses when accessing the Platform UI has been fully resolved. Root Cause: We identified a configuration issue within our load balancers that was incorrectly routing authentication requests. Resolution: The configuration has been corrected, and traffic is flowing normally. All systems are now fully operational, and data processing remained unaffected throughout the incident. Thank you for your patience as we sorted this out.

Read the full incident report →

Started Jun 30, 2026, 10:42 AM UTC· resolved Jun 30, 2026, 11:15 AM UTC

1. investigating Jun 30, 2026, 10:42 AM UTC

   We are experiencing elevated error rates within our Authentication service. As a result, users may encounter difficulties logging into the platform or executing specific connector runs. Our engineering team is actively investigating the root cause and working toward a resolution.

2. resolved Jun 30, 2026, 11:15 AM UTC

   Dear customers, the issue has been identified as defective hardware on one of our database nodes in the service cluster. The faulty node has been isolated and authentication is now working as usual.

Read the full incident report →

Started Jun 02, 2026, 10:28 AM UTC· resolved Jun 02, 2026, 11:26 AM UTC

1. investigating Jun 02, 2026, 10:28 AM UTC

   We are currently experiencing issues with the Platform Authentication System. Data Processing is not affected. We are currently investigating the issue, please bear with us in the meantime.

2. identified Jun 02, 2026, 10:31 AM UTC

   The issue proved to be only transitory, and the Platform is working again. We will post a resolution note when the root cause has been identified.

3. resolved Jun 02, 2026, 11:26 AM UTC

   Between 10:21 and 10:28 UTC on June 2, access to Productsup Platform was briefly unavailable for all users. The cause was a short replication pause on a backend database cluster, which delayed authentication requests for about 6 minutes. Service recovered automatically and login is working normally again. No action is needed on your side. We're reviewing the backend database behaviour to prevent a recurrence.

Read the full incident report →

Started Jun 01, 2026, 11:51 AM UTC· resolved Jun 02, 2026, 04:54 AM UTC

1. investigating Jun 01, 2026, 11:51 AM UTC

   We are currently experiencing issues with a long-distance fibre link to our Frankfurt location. As a precaution, we have temporarily taken this connection offline to maintain network stability. Traffic is automatically being rerouted via alternative transit and peering paths. No service interruption is expected. We are monitoring the situation and will restore the link once stability is confirmed.

2. identified Jun 01, 2026, 02:31 PM UTC

   We have identified the root cause of the network instability affecting our Frankfurt long-distance link. The issue has been confirmed to originate with our upstream fibre supplier, who is currently working to implement a fix. The affected link remains offline while remediation is in progress. Traffic continues to be routed via redundant transit and peering paths. We will provide a further update once the fix has been applied and the link is restored.

3. monitoring Jun 01, 2026, 02:49 PM UTC

   Our supplier has confirmed that the underlying issue has been resolved. We have re-enabled the Frankfurt link and are closely monitoring its stability.

4. resolved Jun 02, 2026, 04:54 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 29, 2026, 03:32 PM UTC· resolved May 29, 2026, 06:42 PM UTC

1. investigating May 29, 2026, 03:32 PM UTC

   We are currently investigating delays affecting connector runs. Some runs may take longer than usual to start or resume. No data is at risk, and runs will continue to process — they may simply be slower to complete during this period. Our engineering team is actively working to restore normal processing times. We will provide an update when this happens.

2. resolved May 29, 2026, 06:42 PM UTC

   This incident has been resolved. Data source imports are processing normally again. Earlier today, some imports experienced longer-than-usual processing times, primarily affecting connections to third-party platforms that were applying API rate limits to incoming requests. No data was lost, and affected imports were retried automatically. The backlog has now cleared and import times have returned to normal. We have also taken steps to reduce the likelihood of similar delays in the future. Thank you for your patience.

Read the full incident report →

Started Apr 07, 2026, 01:09 PM UTC· resolved Apr 07, 2026, 02:30 PM UTC

1. investigating Apr 07, 2026, 01:09 PM UTC

   We are currently experiencing issues with the platform. Our team is actively investigating the problem and working to identify the root cause. We will provide further updates as soon as more information becomes available.

2. monitoring Apr 07, 2026, 01:41 PM UTC

   The issue was identified and has been resolved. The platform is now operational. Our team will continue to closely monitor the systems to ensure stability and complete resolution of the incident.

3. resolved Apr 07, 2026, 02:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 03, 2026, 01:49 PM UTC· resolved Mar 03, 2026, 01:49 PM UTC

1. resolved Mar 03, 2026, 01:49 PM UTC

   Elevated errors on a network interface in one of our storage cluster nodes caused disruption on our Transport File Service (HTTP/FTP). After the faulty node was isolated by our engineer on duty, downloads were available as normal.

Read the full incident report →

Started Feb 03, 2026, 07:22 AM UTC· resolved Feb 03, 2026, 09:56 AM UTC

1. investigating Feb 03, 2026, 07:22 AM UTC

   Dear customers, we're currently investigating some database cluster issues that may have caused some jobs to fail. We will update this incident when the root cause will be identified. Your Productsup Tech Operations Team

2. identified Feb 03, 2026, 07:32 AM UTC

   We have identified the root cause of the incident. A backup node rejoined the cluster unexpectedly, which triggered networking configuration conflicts preventing other nodes from connecting to it. This caused a brief cluster-wide pause (~2 minutes) while the state auto-resolved. Services are now operating normally. We are monitoring the situation and will provide further updates if needed.

3. resolved Feb 03, 2026, 09:56 AM UTC

   The networking configuration conflict caused by the backup node rejoining the cluster has been fully resolved. The cluster auto-recovered and all nodes are operating normally. No data loss or lasting impact. We will review the backup node rejoin behavior and networking config to prevent recurrence. Thanks for your patience.

Read the full incident report →

Started Jan 21, 2026, 03:04 PM UTC· resolved Jan 21, 2026, 03:57 PM UTC

1. monitoring Jan 21, 2026, 03:04 PM UTC

   Incident: FTP Download Service Disruption Status: Monitoring Duration: ~30-60 minutes (ended 17:34 UTC) Impact: Customers experienced failures when downloading files via FTP during this window. Root Cause: Internal network maintenance window on FTP infrastructure resulted in connectivity issues affecting the service. Resolution: Network changes were reverted and full service restored. The FTP fleet is currently operating normally with enhanced monitoring active. What We're Doing: - Continuous monitoring of FTP service health and network metrics - Post-incident review to improve change management procedures for infrastructure maintenance We apologize for the disruption and appreciate your patience.

2. resolved Jan 21, 2026, 03:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 18, 2025, 02:04 PM UTC· resolved Dec 18, 2025, 03:00 PM UTC

1. investigating Dec 18, 2025, 02:04 PM UTC

   We are currently investigating an issue affecting our image service. Some users may experience broken images or timeouts during generation. Our team is actively working to identify the root cause. While not all requests are impacted, we appreciate your patience as we work toward a full resolution. We will provide further updates as soon as more information is available.

2. resolved Dec 18, 2025, 03:00 PM UTC

   We have successfully resolved the issues affecting image generation. The service is now operating at full capacity. Our investigation identified that the timeouts were caused by an isolated surge in volume that exceeded our standard system limits. To prevent this from recurring, we have increased our capacity safeguards. These updates will ensure consistent performance for all users, even during periods of high demand. Thank you for your patience while we stabilized the service.

Read the full incident report →

Started Sep 24, 2025, 11:33 AM UTC· resolved Sep 24, 2025, 11:50 AM UTC

1. investigating Sep 24, 2025, 11:33 AM UTC

   Dear customers, our platform is currently above load, which may cause the interface to be slower than usual, and some jobs to be waiting in queue. We are currently investigating the issue. Please bear with us as our team is committed to resolve this incident as soon as possible. Best regards, Your Productsup Technical Operations Team

2. identified Sep 24, 2025, 11:36 AM UTC

   Dear customers, we have linked the performance issue to a defective fiber optic transceiver in our Falkenstein datacenter. Our provider is currently replacing the module. Thanks for your patience as the incident should be resolved shortly.

3. resolved Sep 24, 2025, 11:50 AM UTC

   Dear customers, the module has been replaced by our provider and performance is back to normal. Thanks again for your patience during this incident. Best regards, Your Productsup Tech Operations Team

Read the full incident report →

Started Sep 09, 2025, 11:53 AM UTC· resolved Sep 09, 2025, 04:30 PM UTC

1. investigating Sep 09, 2025, 11:53 AM UTC

   Dear customers, We are seeing elevated numbers of full processing runs, instead of runs that should process the delta of changed products. We are currently investigating the issue. Best regards, Your Productsup Tech Operations Team

2. investigating Sep 09, 2025, 12:46 PM UTC

   We are continuing to investigate this issue.

3. identified Sep 09, 2025, 12:48 PM UTC

   Dear customers, the issue has been identified, our storage cluster which stores Site Workspaces is currently in a failed state. Due to this, you may encounter workspace errors which can be safely ignored. Delta processing runs and parallel processing may not work at all, while full processing runs will succeed nevertheless. Our infrastructure team is working to restore the cluster to a full state as soon as possible, and we'll keep you posted if there are any new developments. Thank you for your understanding, Your Productsup Tech Operations Team

4. monitoring Sep 09, 2025, 01:17 PM UTC

   Dear customers, The cluster has been rebooted into a working state. Delta operations, parallel processing and export2datasource features should function again normally. We are continuing to monitor the restoration process and will debrief this incident later on. Thanks again for your patience and continued support, Your Productsup Operations Team

5. resolved Sep 09, 2025, 04:30 PM UTC

   This incident has been fully resolved. All systems are operating normally. We will continue monitoring the infrastructure closely over the coming days to ensure continued stability.

6. postmortem Sep 09, 2025, 04:31 PM UTC

   # Service Incident Notification - Ceph Storage Cluster \(Processing\) **Date:** September 9, 2025 **Incident Reference:** CEPH-2025-009 **Status:** RESOLVED ## What Happened On September 9, 2025, our storage cluster experienced a service disruption while implementing a planned infrastructure upgrade to enable multi-datacenter operations. During the activation of "stretch mode" \(a feature that allows data to be synchronized across multiple datacenters for improved disaster recovery\), the cluster's internal authentication system encountered an unexpected failure. ## Customer Impact **Duration:** 12:30 → 15:00 \(2hr30m\) **Affected Services:** Data Processing **Data Safety:** All customer data remained secure and intact throughout the incident. No data was lost or corrupted. During the incident period, customers may have experienced: * Temporary inability to access stored workspaces or export2datasource * Intermittent connectivity issues with applications using the storage service ## Root Cause The issue occurred when our storage system automatically reconfigured all data pools during the multi-datacenter setup process. This reconfiguration inadvertently affected critical system components responsible for user authentication, preventing normal access to the cluster even though the underlying data remained safe and accessible. ## Resolution Our engineering team worked to restore service by upgrading the storage cluster software to a newer version that provided additional recovery options not available in the previous version. This upgrade allowed us to safely reverse the multi-datacenter configuration and restore normal cluster operations. ## What We're Doing to Prevent This * **Enhanced Testing:** We are improving our testing environments to better replicate production conditions and catch similar issues before they affect live services * **Software Version Management:** We are updating our deployment standards to use newer software versions that provide better recovery capabilities for major configuration changes * **Monitoring Improvements:** We are implementing additional real-time monitoring for authentication systems during major infrastructure changes * **Rollback Procedures:** We are developing more comprehensive rollback procedures for complex infrastructure modifications ## Our Commitment We sincerely apologize for any inconvenience this incident may have caused. Data security and service reliability are our highest priorities. We are committed to learning from this experience and implementing the necessary improvements to prevent similar incidents in the future. If you have any questions or concerns about this incident, please contact our support team.

Read the full incident report →

Started Aug 11, 2025, 02:20 PM UTC· resolved Aug 11, 2025, 05:33 PM UTC

1. investigating Aug 11, 2025, 02:20 PM UTC

   Dear customers, We're observing longer queues than normal in our processing system. We’re currently investigating and will have more information shortly. Thank you for your patience.

2. identified Aug 11, 2025, 02:48 PM UTC

   Dear customers. we have performed a restart of our processing scheduler systems in order to unblock the queue. All systems are now back to normal state and jobs should be processing in due time. We will continue to monitor the system in case any further issues arise. Thank you for your understanding, Your Productsup Technical Operations Team

3. resolved Aug 11, 2025, 05:33 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 24, 2025, 07:41 PM UTC· resolved Jun 24, 2025, 07:55 PM UTC

1. identified Jun 24, 2025, 07:41 PM UTC

   Following our investigation into the recent network incident, our engineering team has identified an execution bug affecting several projects. This bug is currently preventing some jobs from starting. To urgently resolve this issue, we must perform a full restart of our data processing scheduler. During this essential maintenance: - All currently executing jobs will be stopped. - No new jobs will be able to start until the executor is restarted. We understand this causes inconvenience and appreciate your patience as we work to restore full functionality. We will provide an update as soon as the restart is complete and services are back to normal.

2. resolved Jun 24, 2025, 07:55 PM UTC

   Dear Customers, We are pleased to inform you that the full restart of our data processing scheduler has been successfully completed. The maintenance proceeded without issues, and the execution bug preventing some jobs from starting has been resolved. All jobs will now restart and resume execution according to their regular schedules. Thank you for your understanding and patience during this essential maintenance. We will continue to monitor the system closely. Sincerely, Productsup Technical Operations Team

Read the full incident report →

Started Jun 05, 2025, 10:47 PM UTC· resolved Jun 06, 2025, 12:30 AM UTC

1. investigating Jun 05, 2025, 10:47 PM UTC

   Due to to the unexpected interruption of our network connection between two of our datacenters, some services might encounter degraded performance or be intermittently unreachable. We are currently investigating what caused the interruption of the connection and will update this thread when we have done so. Best regards, Productsup Network Operations Center

2. resolved Jun 06, 2025, 12:30 AM UTC

   The network connection between our two datacenters has been restored, and services are fully available at this time. Thanks for your patience and continued support. Best regards, Productsup Network Operations Center

Read the full incident report →

Started May 20, 2025, 04:28 PM UTC· resolved May 21, 2025, 06:53 AM UTC

1. investigating May 20, 2025, 04:28 PM UTC

   Due to an influx of jobs that is higher than usual, a queue has built up in our scheduler system. We are actively working to process the backlog and restore normal operations. Our team is monitoring the situation closely, and jobs are being processed as efficiently as possible. We ask for your understanding and patience as we work through this.

2. monitoring May 20, 2025, 05:40 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 21, 2025, 06:53 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 12, 2025, 11:45 AM UTC· resolved May 12, 2025, 01:37 PM UTC

1. investigating May 12, 2025, 11:45 AM UTC

   We're investigating an elevated number of errors when uploading data to "Productsup Server".

2. monitoring May 12, 2025, 12:57 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 12, 2025, 01:37 PM UTC

   This incident has been resolved.

4. postmortem May 13, 2025, 12:39 PM UTC

   **Incident Overview** On May 12, we experienced a temporary service disruption affecting parts of our transport platform. This occurred during a configuration update aimed at improving our system’s routing capabilities. **Root Cause** The disruption was caused by a misconfiguration in our internal storage system. Although the change had passed staging tests, it behaved differently in production, leading to certain requests being misrouted. **Impact** During the incident, some clients were unable to access services hosted on specific subdomains of our transport platform and jobs couldn't write to destination "Productsup server". Normal operation resumed shortly after the change was rolled back. **Preventive Measures** To prevent this issue from recurring, we are: * Improving our deployment validation processes. * Enhancing staging to more closely reflect production conditions. * Adding more automated checks to detect routing issues earlier. **Conclusion** The issue was resolved by reverting the configuration. Processes have been updated to prevent recurrence. All systems are operating normally.

Read the full incident report →

Started Mar 17, 2025, 11:46 AM UTC· resolved Mar 17, 2025, 01:42 PM UTC

1. identified Mar 17, 2025, 11:46 AM UTC

   Dear Customers, Please be advised that we are currently experiencing degraded transfer and processing speeds due to a network equipment issue, specifically a failed router, in one of our data centers. This has been ongoing since 20:00 Berlin time yesterday. Our provider is scheduled to replace the affected hardware today. We will provide another update once the issue is resolved and performance returns to normal.

2. resolved Mar 17, 2025, 01:42 PM UTC

   Dear customers, our provider has replaced the affected hardware on schedule, and we are observing normal transfer speeds, therefore we are closing this incident. Thank you for your continued support, Your Productsup Technical Operations Team

Read the full incident report →

Started Mar 05, 2025, 07:52 AM UTC· resolved Mar 05, 2025, 08:55 AM UTC

1. investigating Mar 05, 2025, 07:52 AM UTC

   We are currently experiencing a hardware failure on one of our routers. As all our routers are fully redundant, there is no impact on our services. We will be performing a hardware replacement shortly to restore full redundancy. Further updates will be provided as soon as more information is available. Thank you for your understanding. Best regards, Productsup Network Operations Center

2. identified Mar 05, 2025, 07:56 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 05, 2025, 08:22 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 05, 2025, 08:55 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 20, 2025, 08:46 PM UTC· resolved Jan 20, 2025, 08:46 PM UTC

1. resolved Jan 20, 2025, 08:46 PM UTC

   Summary On 17.01.2024, one of the processing storage cluster servers rebooted unexpectedly due to a configuration issue with the object storage gateway. This incident was compounded by previous changes made to accommodate a new replication zone in Frankfurt, leading to operational complications. Incident Details Background Several months ago, modifications were made to include a new replication zone in Frankfurt. However, it appears the toolset used for this configuration did not execute correctly, resulting in the retention of a "default" zone and a "default" zone group alongside the active "fsn" and "fsn-1" zones and zone groups in Falkenstein. Incident Timeline Reboot: The server rebooted unexpectedly. Configuration Issue: Upon investigation, it was found that the object storage gateway connected to the default zone instead of the intended configuration. Service Behavior: Despite the service starting, it encountered issues as there was no data in the default zone, leading to complaints about missing data for incoming requests. Monitoring Oversight: The monitoring system only tracked whether the service was "up," which did not alert us to the underlying issue until a user report was received. Resolution The issue was identified after user reports highlighted the data unavailability. Immediate actions were taken to correct the object storage gateway configuration, ensuring it points to the correct zone. Lessons Learned Monitoring Improvements: To prevent similar incidents in the future, we will enhance our monitoring to include error rates on this cluster. This will allow us to detect unusual patterns that may indicate configuration mistakes or other issues. Configuration Checks: A review of the configuration process for replication zones will be conducted to ensure that toolsets function as expected and do not leave behind unintended configurations.

Read the full incident report →

Started Jan 09, 2025, 03:31 AM UTC· resolved Jan 09, 2025, 07:59 AM UTC

1. investigating Jan 09, 2025, 03:31 AM UTC

   The platform is currently not available. We're investigating the root cause.

2. monitoring Jan 09, 2025, 03:56 AM UTC

   Our main database cluster caused the applications to fail. The root cause has been identified and fixed and all applications resumed normal operations. We'll monitor the situation.

3. monitoring Jan 09, 2025, 03:58 AM UTC

   We are continuing to monitor for any further issues.

4. monitoring Jan 09, 2025, 03:58 AM UTC

   We are continuing to monitor for any further issues.

5. resolved Jan 09, 2025, 07:59 AM UTC

   All systems operating normally.

Read the full incident report →

Started Dec 18, 2024, 09:56 AM UTC· resolved Dec 18, 2024, 04:46 PM UTC

1. investigating Dec 18, 2024, 09:56 AM UTC

   Dear Customers, We are currently experiencing unusually high demand for our software due to a combination of seasonal factors. As a result, some sites may experience delays or, in some cases, become stuck in the queue. Our team is actively investigating all possible root causes to address these challenges. Please rest assured that we are monitoring the situation closely and taking every step to enhance performance. Your experience is important to us, and we appreciate your patience as we work through this. Thank you for your understanding.

2. identified Dec 18, 2024, 12:38 PM UTC

   Dear customers, We have identified a potential issue affecting our service performance. A network equipment component was found to be possibly defective. With the assistance of our data center provider, we have replaced this equipment, which has resulted in significantly higher transfer rates. Additionally, we have upgraded the bandwidth on this equipment to better accommodate these improved speeds. Most customers should start experiencing enhanced times for site runs effective immediately. We are continuing to monitor the entire platform for any other factors that might cause slowness or delays, and we will keep you updated on any further developments. Thank you for your continued patience and support.

3. resolved Dec 18, 2024, 04:46 PM UTC

   Dear Customers, We are pleased to inform you that, according to our monitoring, we are satisfied that the issue has been resolved. We are now experiencing improved site run times and enhanced upload/download performance. Thank you for your patience during this process. We remain committed to providing you with the best possible service and will continue to monitor the situation closely. To prevent similar issues in the future, we are implementing the following measures: Increased Bandwidth Planning: We will reassess and adjust our bandwidth capacity regularly to ensure it meets the demands of our customer base, especially during peak usage times. Enhanced Monitoring Tools: We are upgrading our monitoring systems to provide real-time insights into network performance and quickly identify any anomalies.

Read the full incident report →

Started Dec 13, 2024, 10:46 AM UTC· resolved Dec 13, 2024, 11:41 AM UTC

1. investigating Dec 13, 2024, 10:46 AM UTC

   Description: We are currently experiencing issues with Image Designer (SVG) generation in our Image Service. This is due to a necessary security upgrade, which has temporarily affected our ability to generate images. Impact: Users are unable to generate SVG images at this time. Next Steps: We are aware of the issue and are actively working on a workaround. Updates will be provided as soon as more information is available. Thank you for your patience as we resolve this matter.

2. resolved Dec 13, 2024, 11:41 AM UTC

   Description: We have successfully rolled back Chromium to an earlier version, and SVG generation is now functioning as expected. Impact: Users can now generate SVG images without any issues. Thank you for your patience during this incident. If you experience any further issues, please reach out to our support team.

3. postmortem Dec 13, 2024, 11:44 AM UTC

   ### Incident Overview On 13.12.2024, an upgrade from Chromium 124 to Chromium 130 was deployed on our servers, impacting the Image Service's ability to generate scalable vector images \(SVGs\). This incident was triggered by our automated security upgrade process, which is in place to maintain the security integrity of our software. ### Timeline * **Upgrade Date:** 13.12.2024 07:00 - Chromium was upgraded from version 124 to version 130. * **Incident Detection:** 13.12.2024 09:00 - Users reported issues with SVG generation. * **Investigation:** 13.12.2024 09:30 - The team began investigating the root cause. * **Rollback:** 13.12.2024 12:00 - Chromium was rolled back to version 124, restoring SVG generation functionality. * **Current Status:** Fix for Chromium 130 is in progress and will be thoroughly tested before redeployment. ### Root Cause The upgrade to Chromium 130 introduced incompatibilities that affected SVG generation. This was an unanticipated consequence of the automated security upgrades, which are designed to keep our systems secure by ensuring all software is up to date. ### Resolution To mitigate the immediate issue, we rolled back Chromium to version 124, which restored the SVG generation functionality. The team is currently working on a fix for version 130 to ensure that it can be deployed without causing similar issues in the future. ### Action Items 1. **Testing Fix for Chromium 130:** Ensure that the upcoming fix is thoroughly tested before any deployment. 2. **Review Upgrade Process:** Assess our automated upgrade process to identify potential improvements that minimize risks associated with future upgrades. 3. **Monitoring Enhancements:** Implement enhanced monitoring to quickly identify and address similar issues in the future. ### Conclusion We recognize the impact that this incident had on our users and are committed to ensuring that our systems remain secure while minimizing disruptions. We appreciate your understanding and patience as we work towards a permanent solution for the Chromium 130 upgrade.

Read the full incident report →