Wisely incident

Segment evaluation delays

Notice Resolved View vendor source →

Wisely experienced a notice incident on December 16, 2023 affecting Guest Data Platform (GDP), lasting 1d 23h. The incident has been resolved; the full update timeline is below.

Started
Dec 16, 2023, 11:56 PM UTC
Resolved
Dec 18, 2023, 10:57 PM UTC
Duration
1d 23h
Detected by Pingoru
Dec 16, 2023, 11:56 PM UTC

Affected components

Guest Data Platform (GDP)

Update timeline

  1. investigating Dec 16, 2023, 11:56 PM UTC

    We are experiencing database issues affecting the processing of of some GDP events, guest data updates will be delayed for segments / guest data profiles

  2. investigating Dec 17, 2023, 03:44 AM UTC

    We are still investigating this issue.

  3. identified Dec 17, 2023, 02:22 PM UTC

    We have identified that the issue is with specific subsets of event data that cause segment processing to fail overall. We are attempting to suppress those subsets while bringing in other event data to at least partially restore segment data.

  4. monitoring Dec 17, 2023, 04:11 PM UTC

    We have added code to filter the problem event data and have brought up segment evaluation for everything else. Segment updates are coming up to current. We are separately coordinating a response plan for how to handle the problem data. Most customers should see that their segment data is now catching up. We will notify again when we observe that data is fully current.

  5. monitoring Dec 17, 2023, 04:39 PM UTC

    Data is now current for most customers. We have identified the customers who are still impacted and will be reaching out to them individually to apprise them of the situation. Individual customers will be notified by their CSM as well as their data is restored.

  6. resolved Dec 18, 2023, 10:57 PM UTC

    We have an identified action plan to address data cleanup for impacted customers. We have reached out to those customers individually to notify them. Further communications will be handled through customer support channels going forward.