Treasure Data Outage History

Started Nov 25, 2025, 03:07 PM UTC· resolved Nov 25, 2025, 03:07 PM UTC

1. monitoring Nov 25, 2025, 03:07 PM UTC

   Nov 25, 07:07 PST Completed - The scheduled maintenance has been finished, with upgrades to all sites finished. Nov 25, 06:47 PST Verifying - All sites have been upgraded. We're currently verifying the upgrade and monitoring for regressions. Nov 25, 06:32 PST Update - All sites except for the environment in Korea have been upgraded, and we're verifying the status and monitoring for regressions. Nov 25, 06:06 PST Update - Maintenance is still in progress, pending the resource creation by the infrastructure provider, which is taking longer than expected, but is near completion. Nov 25, 05:39 PST Update - Maintenance is still in progress, pending the resource creation by the infrastructure provider, which is taking longer than expected. We will keep updating this page every 30 minutes. Nov 25, 05:01 PST Update - Maintenance is still in progress, but the completion is delayed as we await resource creation on our infrastructure provider, which is taking longer than expected. Nov 25, 04:35 PST Update - Maintenance is still in progress. Nov 25, 04:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates every 30 minutes, or as necessary. Nov 25, 02:57 PST Update - The scheduled maintenance window for the databases supporting PrecisionML will begin in 1h, from 12:00-13:00 UTC, today (November 25th, 2025). Oct 28, 10:00 PDT Scheduled - On November 25th, 2025, from 12:00-13:00 UTC, the databases supporting PrecisionML will undergo a short maintenance period. We'll be upgrading the databases supporting the PrecisionML feature of the Treasure Data Platform. This upgrade is necessary to apply the latest available performance and security patches. During the maintenance window, customers may experience momentary failures or have PrecisionML runs not created or executed. Any failures will be automatically retried from Treasure Workflow, and no customer action is required to ensure all runs complete. # Communication Beyond this notice, we will provide updates approximately 1 hour before the beginning of the maintenance window, at the start and completion of the operation, and once verification is complete. At that time, all systems will have returned to full functionality, and the Scheduled Maintenance will be closed. If you have any questions or concerns about this maintenance, please feel free to reach out to our Support team at [email protected].

Read the full incident report →

Started Nov 18, 2025, 06:00 AM UTC· resolved Nov 18, 2025, 06:00 AM UTC

1. monitoring Nov 18, 2025, 06:00 AM UTC

   Nov 17, 22:00 PST Completed - The scheduled maintenance has been completed. Nov 17, 19:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov 5, 01:54 PST Update - We will be undergoing scheduled maintenance during this time. Oct 12, 23:53 PDT Scheduled - We are pleased to inform you that we will upgrade our DataTank 2.0 PostgreSQL database engine to version 14.18. This upgrade is part of our commitment to maintaining the highest levels of performance, security, and reliability for your data. *Key benefits of upgrading to version 14.18: - Enhanced performance: Improvements that boost query efficiency and overall system responsiveness. - Increased security: Security patches and updates to protect your data - New features: access to new features and functionalities of PostgreSQL *Upgrade details: - Expected Downtime: 10 minutes per instance. We aim to make this transition as smooth as possible. Our team will be monitoring the process closely to ensure minimal disruption. If you have any questions or concerns, please don’t hesitate to contact our support team. During the maintenance window, the databases will experience a short failover within 120 seconds, after which the respective systems are expected to recover in approximately 10 minutes. During the failover period, customers will observe that the DataTank 2.0 database system might not function properly

Read the full incident report →

Started Nov 12, 2025, 07:00 AM UTC· resolved Nov 12, 2025, 07:00 AM UTC

1. monitoring Nov 12, 2025, 07:00 AM UTC

   Nov 11, 23:00 PST Completed - The scheduled maintenance has been completed. Nov 11, 19:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov 5, 01:49 PST Update - We will be undergoing scheduled maintenance during this time. Oct 12, 23:46 PDT Update - We will be undergoing scheduled maintenance during this time. Oct 12, 23:46 PDT Scheduled - We are pleased to inform you that we will upgrade our DataTank 2.0 PostgreSQL database engine to version 14.18. This upgrade is part of our commitment to maintaining the highest levels of performance, security, and reliability for your data. *Key benefits of upgrading to version 14.18: - Enhanced performance: Improvements that boost query efficiency and overall system responsiveness. - Increased security: Security patches and updates to protect your data - New features: access to new features and functionalities of PostgreSQL *Upgrade details: - Expected Downtime: 10 minutes for your instance We aim to make this transition as smooth as possible. Our team will be monitoring the process closely to ensure minimal disruption. If you have any questions or concerns, please don’t hesitate to contact our support team. During the maintenance window, the databases will experience a short failover within 120 seconds, after which the respective systems are expected to recover in approximately 10 minutes. During the failover period, customers will observe that the DataTank 2.0 database system might not function properly

Read the full incident report →

Started Nov 05, 2025, 07:58 AM UTC· resolved Nov 05, 2025, 07:58 AM UTC

1. monitoring Nov 05, 2025, 07:58 AM UTC

   Nov 4, 23:58 PST Resolved - Services have been restored to normal operations, and all of our systems are reporting healthy once again. If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support Nov 4, 19:55 PST Monitoring - A fix has been implemented and we are monitoring the results. Nov 4, 19:49 PST Investigating - We have encountered a temporary issue with the availability of Premium Audit Logs for Artificial Intelligence services (AI Agent Foundry and Audience Agent) during the period of November 4th 23:19:26 UTC to November 5th 00:57:55 UTC. While some data during this period may not have been captured, our engineering team is actively investigating the situation. We are taking all necessary steps to restore full service and ensure that all future logs are recorded without issue. We apologize for any inconvenience this may have caused and will keep you updated on our progress.

Read the full incident report →

Started Oct 20, 2025, 11:15 PM UTC· resolved Oct 20, 2025, 11:15 PM UTC

1. monitoring Oct 20, 2025, 11:15 PM UTC

   Oct 20, 16:15 PDT Resolved - Services have been restored to normal operations, and all of our systems are reporting healthy once again. If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support Oct 20, 14:42 PDT Update - Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event. We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops. Oct 20, 12:35 PDT Monitoring - We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up. Our team is continuing to track recovery closely, and we'll inform you of any new developments. Oct 20, 11:49 PDT Update - We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues. At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys. You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon. Oct 20, 08:05 PDT Update - Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0. As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider. Oct 20, 01:27 PDT Investigating - We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services. We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start. Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Read the full incident report →

Started Oct 20, 2025, 09:32 PM UTC· resolved Oct 20, 2025, 09:32 PM UTC

1. monitoring Oct 20, 2025, 09:32 PM UTC

   Oct 20, 14:32 PDT Resolved - Delays should be improving for all customers, and services should be returning to normal. Given the similarity of the two issues, we'll continue tracking any concerns related to this interruption as part of the "[US Region] Issue Affecting Treasure Data Services" issue. For further updates, please track here: https://status.treasuredata.com/incidents/w7v5wtmkdllj If you experience continued issues specifically relating to our Profiles API or SSO Logins, please contact our support team: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support Oct 20, 11:53 PDT Identified - We are continuing to work with our service provider to track this issue. Customers may continue to observe higher latency than normal and may notice delays in processing. We are monitoring the situation closely and are adding additional capacity in anticipation of a recovery. As this situation develops, we will keep you updated on any changes in expected behavior. Oct 20, 08:09 PDT Update - Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. SSO login is working fine, however profile API 1.0 is still recovering its service with latency. As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider. Oct 20, 03:15 PDT Update - We are also investigating reports indicating that this incident might be causing login errors for customers using SSO (Single Sign-On) to log into our service. Customers attempting a new login via SSO may be unable to access the service. We believe users who are already logged in (have an active session) and users logging in with methods other than SSO (e.g., username/password) are not affected by this login issue. We are working to confirm this. Oct 20, 01:53 PDT Investigating - We are aware of an issue affecting log writing and reading for the Profile API in all regions. This is due to an incident with an upstream service. Our team is monitoring the provider's status. We will provide an update shortly.

Read the full incident report →

Started Oct 17, 2025, 01:55 AM UTC· resolved Oct 17, 2025, 01:55 AM UTC

1. monitoring Oct 17, 2025, 01:55 AM UTC

   Oct 16, 18:55 PDT Resolved - After a successful monitoring period, we can confirm that our fix is working and the issue is now resolved. Thank you for your patience. Oct 15, 22:05 PDT Monitoring - We've identified that an internal component between our query engine which extracts the number of records that match Segment rules and Audience Studio ran into a capacity issue during a specific time. We have increased the capacity and will continue to monitor the situation. If the issue does not reoccur by approximately 2025/10/17 01:00 (+00:00), we will consider this incident resolved and close this. Oct 15, 11:16 PDT Investigating - We are currently investigating an issue where some customers are experiencing intermittent “query expired” errors when retrieving segment counts in Audience Studio. In many cases, queries may succeed after a retry, but this behavior can impact the ability to view or use segment results reliably. Our engineering teams are actively investigating the root cause and working to mitigate the issue. We will provide updates here as soon as more information becomes available. In the meantime, affected users may be able to work around the issue by retrying the query after a short interval.

Read the full incident report →

Started Oct 15, 2025, 05:06 AM UTC· resolved Oct 15, 2025, 05:06 AM UTC

1. monitoring Oct 15, 2025, 05:06 AM UTC

   Oct 14, 22:06 PDT Resolved - The issue has been fully resolved. Thank you for your patience. Oct 14, 21:38 PDT Monitoring - The issue has been addressed with an applied fix. We are currently monitoring the system to ensure the services are fully restored and stable. Thank you for your patience while we addressed this. Oct 14, 19:52 PDT Identified - The issue has been identified and a fix is being implemented. Oct 14, 19:05 PDT Investigating - We are currently experiencing an issue with email delivery for Treasure Data system emails, such as password resets. The affected email deliveries are: * Default email notification of Treasure Workflow * Password reset * New user invitation * Password change * Email address change * User lock notifications The following emails are NOT affected: * Email delivery with the mail>: operator of Treasure Workflow * Email delivery with Engage Studio Our team is actively working to resolve this issue. We sincerely apologize for any inconvenience or disruption this may cause.

Read the full incident report →

Started Sep 30, 2025, 03:37 AM UTC· resolved Sep 30, 2025, 07:22 AM UTC

1. investigating Sep 30, 2025, 03:37 AM UTC

   We are currently investigating an issue where AI agents in the AI Agent Foundry may prematurely stop their response without providing a final conclusion, especially during their thinking process. This behavior does not generate an error message. Our initial findings show it is more likely to occur in agents handling long or extensive conversations, though we have not yet identified a specific threshold. Our engineering team is actively investigating the root cause. We will provide another update as soon as more information is available. We apologize for any inconvenience this may cause.

2. identified Sep 30, 2025, 06:08 AM UTC

   We have an update on the root cause of the issue causing AI agents to provide incomplete responses. Our investigation has found that when an agent uses its tools, the user's original prompt was being sent multiple times to the underlying AI model. This duplication caused the model to incorrectly process the tool results, leading it to return an empty response instead of a complete answer. The hotfix we are preparing will correct this behavior by ensuring the user's prompt is sent only once for each task. This will allow the AI to properly process all tool results and generate a complete, final response. We are working to deploy this fix as quickly as possible and will provide another update once it is ready. Thank you for your continued patience.

3. monitoring Sep 30, 2025, 06:53 AM UTC

   The hotfix for the incomplete agent responses has been deployed. This should prevent the duplicate prompt issue during tool use. We are now monitoring the platform to ensure everything is working as expected. We will resolve this incident once we've confirmed stability.

4. resolved Sep 30, 2025, 07:22 AM UTC

   After a successful 30-minute monitoring period, we can confirm that our fix is working and the issue is now resolved. All AI agents are functioning as expected. Thank you for your patience.

Read the full incident report →

Started Sep 26, 2025, 07:12 PM UTC· resolved Sep 26, 2025, 11:24 PM UTC

1. identified Sep 26, 2025, 07:12 PM UTC

   We are currently experiencing delays in our Profile API replication pipeline in the Tokyo region. This issue began at approximately 13:00 UTC and is affecting all customers in this region. Impact: - Profile API responses may return stale data (up to several hours behind) - Lookup functionality remains available but data may be inconsistent Our engineering team is actively working on recovery. We estimate normal service will be restored within 8-10 hours. We will continue to provide updates as we make progress toward resolution.

2. monitoring Sep 26, 2025, 11:02 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 26, 2025, 11:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 24, 2025, 01:55 AM UTC· resolved Sep 24, 2025, 02:40 AM UTC

1. resolved Sep 24, 2025, 06:53 AM UTC

   We experienced a partial issue on September 24th, between 01:55 and 02:40 UTC, where some users may have encountered incorrect permission errors when accessing their tables and databases in Hive and Trino. The issue was caused by a high load on our table API, which temporarily struggled to retrieve correct permission data. All underlying data remained secure and unaffected. The API load has returned to normal, and all services are operating as expected. If you had any jobs that failed with a permission error during this period, please try running them again. We apologize for any confusion this may have caused.

Read the full incident report →

Started Aug 18, 2025, 09:03 PM UTC· resolved Aug 18, 2025, 09:49 PM UTC

1. investigating Aug 18, 2025, 09:03 PM UTC

   Treasure Data Insights is currently unavailable; we are investigating this issue.

2. resolved Aug 18, 2025, 09:49 PM UTC

   The issue was resolved; it was linked to a vendor issue. The vendor has resolved the problem that was impacting TD Insights.

Read the full incident report →

Started Jul 02, 2025, 02:53 AM UTC· resolved Jul 02, 2025, 08:03 AM UTC

1. investigating Jul 02, 2025, 02:53 AM UTC

   We are currently investigating a slowdown affecting Treasure Insight on the AP02 site. As a result, you may experience longer than usual processing times, and some requests may time out or fail.

2. monitoring Jul 02, 2025, 07:17 AM UTC

   We identified the cause of the performance degradation, and we fixed it. We confirmed our metrics are back to normal. We're keeping the monitor.

3. resolved Jul 02, 2025, 08:03 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 23, 2025, 06:45 AM UTC· resolved Jun 23, 2025, 07:58 AM UTC

1. identified Jun 23, 2025, 06:45 AM UTC

   We are currently experiencing an issue with the CDP API concerning the retrieval of realtime attributes. When opening the settings screen or attempting to change settings, realtime attribute values cannot be retrieved, resulting in an error. This issue exclusively affects the realtime segment settings screen and does not impact the actual behavior of realtime segments. Our team is actively working to resolve this issue. We sincerely apologize for any inconvenience or disruption this may cause.

2. resolved Jun 23, 2025, 07:58 AM UTC

   We are writing to inform you that the issue with realtime attribute retrieval in the CDP API occurred on June 23, 2025, at 04:46 UTC and was resolved on June 23, 2025, at 07:55 UTC. We sincerely apologize for any inconvenience this may have caused.

Read the full incident report →

Started Jun 01, 2025, 04:17 PM UTC· resolved Jun 02, 2025, 05:43 PM UTC

1. investigating Jun 01, 2025, 04:17 PM UTC

   Our Ingest API is experiencing a performance issue. We are investigating the cause.

2. investigating Jun 01, 2025, 04:35 PM UTC

   Current Status: A fix has been applied and is currently under observation. Impact: An issue was identified with data ingestion(delayed ingestion) between Sunday 8:00 AM PST - 8:30 AM PST. Remediation: A fix has been implemented, and new incoming data is now processing normally. Next Steps: We are actively working to resume sending the affected data, which will arrive out of order along with new incoming data. Updates to Follow: Further details will be provided as the situation progresses.

3. investigating Jun 01, 2025, 05:18 PM UTC

   Latest Update: Our team has deployed a fix, and the system is now processing the backlog of data at a controlled rate. Estimated Time to Full Recovery: ~7 hours We’re actively monitoring the recovery and will provide further updates as progress continues. Thank you for your patience.

4. investigating Jun 02, 2025, 01:49 AM UTC

   Latest Update: Our team is still actively monitoring and assessing the processing of the data backlog. Thank you for your continued patience.

5. monitoring Jun 02, 2025, 04:47 AM UTC

   Latest Update: Our internal graphs show that the overwhelming majority of the data has been processed. There is small amount of residual data that is taking longer than expected to process. We will leave this status page in a Monitoring state until we are certain everything has been processed. Thank you for your continued patience.

6. monitoring Jun 02, 2025, 04:26 PM UTC

   Latest Update: Our team is still actively monitoring and assessing the processing of the data. We will leave this status page in a Monitoring state until we are certain everything has been processed. Thank you for your continued patience.

7. monitoring Jun 02, 2025, 05:37 PM UTC

   We are continuing to monitor for any further issues.

8. resolved Jun 02, 2025, 05:43 PM UTC

   This incident has been resolved. Duration: ~35 minutes of processing delay. Data Ingestion delay between June 1 - 7:35 AM to June 1 - 10:35 AM PST. Affected Customers: All customers ingesting data to AWS US region during the incident window Impact: Delayed data availability in Plazma (up to 40 mins) No data loss occurred

Read the full incident report →

Started Apr 15, 2025, 07:55 AM UTC· resolved Apr 15, 2025, 10:27 AM UTC

1. investigating Apr 15, 2025, 07:55 AM UTC

   Our Trino service is experiencing an issue. We are investigating the cause.

2. identified Apr 15, 2025, 08:09 AM UTC

   The root cause has been identified and we are applying the fix.

3. monitoring Apr 15, 2025, 08:48 AM UTC

   We are observing recovery. We continue to monitor for full recovery.

4. resolved Apr 15, 2025, 10:27 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 09, 2025, 01:24 AM UTC· resolved Apr 09, 2025, 03:22 AM UTC

1. investigating Apr 09, 2025, 01:24 AM UTC

   We are currently investigating an issue where Hive Jobs and the Result Export jobs triggered from Hive Jobs are not functioning properly. Our team is actively looking into the cause of the issue. We will provide updates as soon as more information becomes available.

2. monitoring Apr 09, 2025, 01:49 AM UTC

   We have identified the root cause and applied a hotfix. The issue was that Hive Jobs were unable to start properly, which in turn caused some Result Export jobs triggered from those Hive Jobs to fail. We are currently monitoring the system to ensure that the situation continues to improve.

3. resolved Apr 09, 2025, 03:22 AM UTC

   We have confirmed that Hive jobs are now working properly, and the related Result Export jobs are functioning as expected. The impact occurred between April 8, 16:00 UTC and April 9, 01:35 UTC. If you had any jobs that failed during the affected time window, please re-run them as needed. We sincerely apologize for the inconvenience this my have caused.

Read the full incident report →

Started Feb 11, 2025, 05:48 PM UTC· resolved Feb 11, 2025, 09:47 PM UTC

1. investigating Feb 11, 2025, 05:48 PM UTC

   Our utilization dashboards do not update with up-to-date information. Customers accessing their Treasure Data usage dashboards will see a gap in usage details from early Saturday morning UTC. There is no impact on ongoing Treasure Data usage, and all usage information is correctly stored internally. However, the dashboard where customers can view their consumption is not up-to-date. Note this is a reporting problem only. There is no indication of any issues with regular Treasure Data usage. We are working to diagnose the issue and will provide an update in the next hour.

2. identified Feb 11, 2025, 07:05 PM UTC

   We have identified the cause of this issue and are working to restore service. Once the service is restored, we expect it will take a few hours to catch up on usage data for the last few days.

3. monitoring Feb 11, 2025, 08:34 PM UTC

   We have remediated the issue and are processing usage data from the last 4 days. Users should see the usage dashboards catching up. We expect this process to take about another hour to complete.

4. resolved Feb 11, 2025, 09:47 PM UTC

   This issue is resolved and our utilization dashboards should be showing up-to-date information. If you observe anything unusual in your usage data, please contact our support team. Thank you for your patience while working through this issue.

Read the full incident report →

Started Jan 30, 2025, 10:54 AM UTC· resolved Jan 30, 2025, 03:43 PM UTC

1. investigating Jan 30, 2025, 10:54 AM UTC

   We detected degraded performance of personalization API and an error rate increase. We are currently investigating this issue.

2. monitoring Jan 30, 2025, 11:38 AM UTC

   We are currently observing that the performance degradation and error rate have improved. We continue to closely monitor the metrics.

3. monitoring Jan 30, 2025, 12:31 PM UTC

   We are continuing to monitor for any further issues.

4. monitoring Jan 30, 2025, 02:18 PM UTC

   We are still monitoring the service. Between Thursday, 30 Jan 2025, 10:00 UTC to 11:05 UTC, customers experienced elevated error rates and longer latency for Profiles API lookup. Currently, the cluster workload has calmed down and is operating normally. Our response team is ready to provision additional processing capacity. However, we are closely monitoring the service status to avoid further downtime during peak times. In addition to it, we are working on isolating problematic accesses from the service. We will keep the status page open and update you on the progress.

5. resolved Jan 30, 2025, 03:43 PM UTC

   We implemented fundamental isolation to a problematic configuration at 14:42 UTC. The remediation caused the cluster workload to drop from 60% to 1%. On Friday, we implemented write access isolation to the problematic configuration. It stopped the cluster workload from growing. Today, we implemented read access isolation that restored the cluster workload to the previous level. The system is operating normally now. We close the incident. We acknowledge we need further actions to prevent the same incident from happening again by a similar configuration. We will post further postmortem when we are ready.

Read the full incident report →

Started Jan 23, 2025, 07:34 AM UTC· resolved Jan 23, 2025, 12:18 PM UTC

1. investigating Jan 23, 2025, 07:34 AM UTC

   We are currently observing errors or performance degradation for the personalization API. We are investigating the cause of the issue now.

2. identified Jan 23, 2025, 08:05 AM UTC

   The response team confirmed the symptom is from the same cause as the previous incidents. We are provisioning additional concurrency capacity to the environment. We will update you when it is completed.

3. identified Jan 23, 2025, 10:37 AM UTC

   We provisioned additional capacity at 10:00 am UTC to support the increasing workload. It improved the latency, but we still observed errors and long latency for a small amount of requests. The response team started providing another concurrency capacity. Unlike the previous methods, the new process should not take longer for provisioning. We will update the result in 30 minutes.

4. identified Jan 23, 2025, 11:08 AM UTC

   We successfully provisioned 2x capacity in 30 minutes. New resources improved the latency, but the error rate is still high. The response team is planning to implement another remediation instead of adding resources. We will update you in 30 minutes.

5. monitoring Jan 23, 2025, 12:16 PM UTC

   The response team found problematic real-time segment configurations of one customer's Parent Segment that possibly contributed to consuming the concurrency capacity. The team updated the real-time event routing configuration to mitigate the high latency issue. Combined with capacity addition operations, the team stabilized the Profiles API cache cluster. If you experience any delays or abnormal errors, please reach out to our support team. Thank you for your patience and understanding during this incident. We will update the postmortem with further remediation plan as promised.

6. resolved Jan 23, 2025, 12:18 PM UTC

   Between Thursday, 23 Jan 2025 07:20 UTC to 11:40 UTC, customers experienced elevated error rates and increased latency related to Profiles API. A fix has been implemented, and the issue has been resolved. If you experience any delays or abnormal errors, please reach out to our support team. Thank you for your patience and understanding during this incident. We will share an incident retrospective soon.

Read the full incident report →

Started Jan 20, 2025, 01:34 PM UTC· resolved Jan 20, 2025, 04:04 PM UTC

1. investigating Jan 20, 2025, 01:34 PM UTC

   We have observed that the users are not able to access the Treasure Insights. We are currently investigating the issue.

2. investigating Jan 20, 2025, 02:17 PM UTC

   We are still investigating the issue.

3. investigating Jan 20, 2025, 02:49 PM UTC

   The situation remains the same as it was in the last update.

4. identified Jan 20, 2025, 02:58 PM UTC

   The problem has been identified, and we are currently working on a solution.

5. monitoring Jan 20, 2025, 03:44 PM UTC

   A fix has been implemented, and we are monitoring the service to ensure everything is functioning correctly.

6. resolved Jan 20, 2025, 04:04 PM UTC

   We would like to inform you that the issue has been fully resolved. Incident Impact Details: - The Treasure Insights were returning 502 errors and it was unreachable during the incident. Incident Impact Time: - Start: January 20, 09:51 UTC - End: January 20, 15:35 UTC We apologize for any inconvenience this may have caused and thank you for your patience and understanding.

Read the full incident report →

Started Jan 20, 2025, 10:33 AM UTC· resolved Jan 20, 2025, 11:41 AM UTC

1. investigating Jan 20, 2025, 10:33 AM UTC

   We are currently investigating this issue.

2. monitoring Jan 20, 2025, 10:35 AM UTC

   Through our investigation, we identified the cause of the issue and have applied some remediation. Our team is closely monitoring the system to ensure continued stability.

3. monitoring Jan 20, 2025, 11:10 AM UTC

   We are observing fewer errors now. However, we are still monitoring and re-evaluating the remedial steps to confirm better performance.

4. resolved Jan 20, 2025, 11:41 AM UTC

   We would like to inform you that the issue has been fully resolved. Incident Impact Details: - Personalization API has experienced an outage leading to increased errors and timeouts. Incident Impact Time: - Start: January 20, 07:45 UTC - End: January 20, 11:15 UTC We apologize for any inconvenience this may have caused and thank you for your patience and understanding.

Read the full incident report →

Started Dec 08, 2024, 07:48 PM UTC· resolved Dec 08, 2024, 08:27 PM UTC

1. investigating Dec 08, 2024, 07:48 PM UTC

   Some users may be experiencing degraded performance when running presto or trino jobs. We are investigating the incident. At present all users in the EU central region may be affected.

2. monitoring Dec 08, 2024, 08:05 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 08, 2024, 08:27 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 13, 2024, 07:09 AM UTC· resolved Nov 13, 2024, 10:16 AM UTC

1. investigating Nov 13, 2024, 07:09 AM UTC

   Since approximately 4:00 UTC, we have been experiencing an issue with requests to CDP KVS, which may be affecting Profiles API functionality, causing delays in KVS data synchronization and updates to real-time segment information. Our team is actively investigating and working to resolve the issue as quickly as possible. Please note that Realtime 2.0 is not affected.

2. monitoring Nov 13, 2024, 09:17 AM UTC

   Through our investigation, we identified the cause of the issue as a recent release operation. We have reverted all changes from this release, and normal functionality has been restored. Our team is closely monitoring the system to ensure continued stability.

3. resolved Nov 13, 2024, 10:16 AM UTC

   We would like to inform you that the issue has been fully resolved. Incident Impact Details: - Profiles API experienced an increased frequency of errors and timeouts. - The latest logs were not reflected in real-time segments. Incident Impact Time by Region: us: - Start: November 13, 04:14 UTC - End: November 13, 08:55 UTC aws-tokyo: - Start: November 13, 04:14 UTC - End: November 13, 08:54 UTC eu01: - Start: November 13, 04:17 UTC - End: November 13, 08:51 UTC ap02 - Start: November 13, 04:15 UTC - End: November 13, 09:01 UTC ap03 - Start: November 13, 04:17 UTC - End: November 13, 08:52 UTC We apologize for any inconvenience this may have caused and thank you for your patience and understanding.

Read the full incident report →

Started Nov 05, 2024, 06:24 PM UTC· resolved Nov 05, 2024, 07:26 PM UTC

1. investigating Nov 05, 2024, 06:24 PM UTC

   We are investigating a possible problem currently causing escalated error rates from presto queries. We will provide an update as soon as we know more.

2. investigating Nov 05, 2024, 06:49 PM UTC

   We are continuously investigating this issue. For most queries we expect they will succeed after one or more automatic retries

3. monitoring Nov 05, 2024, 07:05 PM UTC

   We have applied a fix. The problem looks to be resolved, but we are continuing to monitor.

4. resolved Nov 05, 2024, 07:26 PM UTC

   Between Nov 5, 17:15 UTC and Nov 5, 18:45 UTC, Some customers experienced delays and errors related to presto. The cause was insufficient capacity, which will be investigated further. A fix has been implemented and the issue has been resolved. We apologize for any inconvenience caused. If you have any questions about it, please contact [email protected]

Read the full incident report →