Treasure Data incident
[All Regions] Elevated error rate for CDP KVS
Treasure Data experienced a major incident on November 13, 2024 affecting CDP Personalization - Lookup API and CDP Personalization - Lookup API and 1 more component, lasting 3h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 13, 2024, 07:09 AM UTC
Since approximately 4:00 UTC, we have been experiencing an issue with requests to CDP KVS, which may be affecting Profiles API functionality, causing delays in KVS data synchronization and updates to real-time segment information. Our team is actively investigating and working to resolve the issue as quickly as possible. Please note that Realtime 2.0 is not affected.
- monitoring Nov 13, 2024, 09:17 AM UTC
Through our investigation, we identified the cause of the issue as a recent release operation. We have reverted all changes from this release, and normal functionality has been restored. Our team is closely monitoring the system to ensure continued stability.
- resolved Nov 13, 2024, 10:16 AM UTC
We would like to inform you that the issue has been fully resolved. Incident Impact Details: - Profiles API experienced an increased frequency of errors and timeouts. - The latest logs were not reflected in real-time segments. Incident Impact Time by Region: us: - Start: November 13, 04:14 UTC - End: November 13, 08:55 UTC aws-tokyo: - Start: November 13, 04:14 UTC - End: November 13, 08:54 UTC eu01: - Start: November 13, 04:17 UTC - End: November 13, 08:51 UTC ap02 - Start: November 13, 04:15 UTC - End: November 13, 09:01 UTC ap03 - Start: November 13, 04:17 UTC - End: November 13, 08:52 UTC We apologize for any inconvenience this may have caused and thank you for your patience and understanding.