Service degradation - Campaign images are deleted when uploaded in Edit mode

Update timeline

1. identified Sep 04, 2024, 10:12 AM UTC

   We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

2. resolved Sep 04, 2024, 11:39 AM UTC

   The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

3. postmortem Sep 05, 2024, 01:57 PM UTC

   Incident Initiation: On September 4, 2024, at 11:59 AM CET, an anomaly was detected in the internal component of Hive, which initiated an incident. The issue specifically affected the campaign designer, causing newly uploaded images to disappear for users. Investigation: The engineering team immediately began investigating the issue. It was discovered that the newly uploaded images were not being transferred to long-term storage due to a malfunction in the storage service. The root cause was identified as a misconfiguration introduced during a recent update to the image handling system. Mitigation and Resolution: A temporary mitigation was put in place to ensure that images uploaded after the detection of the issue were properly stored. Following this, a correction was developed and deployed at 01:44 PM CET to restore the application's functionality. This fix addressed the underlying configuration error, and the system returned to normal operation. Impact and Scope: The incident impact was on across all regions and affected only certain tenants with a specific configuration attempting to upload images during the incident window. Post-Incident Actions: Procedural improvements will be implemented, including enhanced monitoring of storage services. The engineering team is also reviewing the update process to prevent similar misconfigurations.