Platform Service Degradation - Some Users Unable To View Or Edit Studio Campaigns

Update timeline

1. investigating Sep 04, 2024, 01:41 PM UTC

   We are currently investigating reports where some users are unable to view or edit campaigns they have access to. We will provide you with an update within 1 hour.

2. identified Sep 04, 2024, 02:21 PM UTC

   We have identified a potential cause of the service disruption, and are working on mitigating this issue. Another update within 1 hour.

3. identified Sep 04, 2024, 03:10 PM UTC

   We are currently validating the proposed fix for this issue in a staging environment and will deploy it in the production environment upon successful testing. Another update within 1 hour.

4. identified Sep 04, 2024, 04:09 PM UTC

   We are currently deploying the proposed fix in the production environment and will provide another update once this is completed.

5. monitoring Sep 04, 2024, 05:01 PM UTC

   The proposed fix has successfully been deployed in the production environment. Please notify our Customer Support team if any issues persist. We will now be placing the affected services under monitoring for now.

6. resolved Sep 09, 2024, 09:38 PM UTC

   This incident has been resolved.

7. postmortem Sep 09, 2024, 09:38 PM UTC

   **Summary:** On September 4th, 2024, starting at 4:27 AM EDT, reports of Studio users unable to view or edit campaigns in Studio were received. Following a correlation of customer reports and initial troubleshooting, a platform service degradation incident was declared at 9:21 AM EDT, and published on our Status Page at 9:41 AM EDT. **Scope:** The scope of this service degradation was restricted to Studio users with multiple audiences assigned to them. **Impact:** Studio users who had multiple audiences assigned to them were unable to view or edit campaigns during the duration of this incident \(12hrs 46mins\). No scheduled campaigns were affected, and campaign viewing, editing, and publishing processes were not inhibited for other users with no audiences assigned or had a single audience assigned. **Root Cause:** The root cause of this incident was determined to be a regression to a misconfigured platform enhancement policy change intended to improve the efficiency of how user-assigned audiences were queried, which had been released at 12:07 AM EDT as part of the scheduled software release maintenance the same day. **Mitigation:** A rollback of the offending policy change was performed and completed by 12:53 PM EDT to restore access to Studio campaigns. **Recurrence Prevention:** To prevent this incident from recurring, we will perform the following actions before releasing the platform enhancement policy change in the future: * Review and correct any misconfiguration on the platform enhancement policy change code. * Add more unit test cases to cover multiple audiences on the modified queries. * Add more regression test cases to cover users with multiple audiences.