Visually.io Outage History

Started Sep 01, 2026, 03:31 PM UTC· resolved Sep 01, 2026, 03:59 PM UTC

1. investigating Sep 01, 2026, 03:31 PM UTC

   Allocator API went down.

2. investigating Sep 01, 2026, 03:32 PM UTC

   Recommendations API went down.

3. investigating Sep 01, 2026, 03:39 PM UTC

   Allocator API recovered.

4. investigating Sep 01, 2026, 03:42 PM UTC

   Degraded Performance – Google Cloud Incident We are experiencing degraded service performance due to an ongoing global Google Cloud Platform (GCP) incident affecting the us-central1 region. Some users may experience increased latency, intermittent errors, or temporary service unavailability. Our team is actively monitoring the situation and working to minimize the impact. We will provide another update as soon as more information becomes available.

5. resolved Sep 01, 2026, 03:59 PM UTC

   Resolved – Google Cloud Incident Google has resolved the issue, and our core smoke tests confirm that all services are operating normally. We’ll continue monitoring Google Cloud for any further developments.

6. resolved Sep 01, 2026, 03:59 PM UTC

   **Resolved – Google Cloud Incident** Google has resolved the issue, and our core smoke tests confirm that all services are operating normally. We’ll continue monitoring Google Cloud for any further developments.

Read the full incident report →

Started Jul 17, 2026, 08:30 AM UTC· resolved Jul 17, 2026, 04:23 PM UTC

1. investigating Jul 17, 2026, 08:30 AM UTC

   Due to a sudden Shopify permissions issue, we’re temporarily unable to sync orders with Visually. We’re actively working with Shopify to resolve it. No data will be lost, and all conversions will be calculated automatically once the issue is resolved.

2. investigating Jul 17, 2026, 03:43 PM UTC

   Shopify permissions have been restored, and orders are now syncing properly. Our team is working to ensure all missing data is populated retroactively.

3. resolved Jul 17, 2026, 04:23 PM UTC

   This incident has been resolved. Shopify permissions have been restored, orders are syncing properly, and all missing data has been populated retroactively.

Read the full incident report →

Started Jul 13, 2026, 01:52 PM UTC· resolved Jul 13, 2026, 02:12 PM UTC

1. investigating Jul 13, 2026, 01:52 PM UTC

   Allocator API went down.

2. resolved Jul 13, 2026, 02:12 PM UTC

   Root Cause Analysis — July 13, 2026 Incident summary On July 13, 2026, we experienced two service interruptions related to the same underlying issue: 07:37–08:36 UTC 13:52–14:12 UTC Root cause: The incident began when our Kubernetes cluster stopped forwarding logs due to an issue with Fluentd, our log collection service. To restore log forwarding, we initiated a non-disruptive upgrade of the Kubernetes node pools. Based on the expected behavior of this procedure, the upgrade was not anticipated to affect service availability. When the upgrade reached the node pool hosting ScyllaDB, Fluentd began processing a backlog of accumulated logs. This caused an unexpected increase in CPU and memory consumption on the affected nodes. As a result, the cluster temporarily lacked sufficient resources to schedule the ScyllaDB pods, leading to service unavailability. Once the node-pool upgrade had started, the operation needed to complete before we could safely apply further changes, this extended the recovery time by approximately 20 minutes. During recovery, we developed a remediation plan to increase the resources available to the ScyllaDB workloads. While deploying that change, our Terraform controller reconciled the environment to its previous configuration and reverted part of the Fluentd update before the remediation was fully applied. This caused the second service interruption. Resolution and preventive measures: We have implemented a permanent remediation that: - Provides additional CPU and memory headroom for all database workloads. - Implementing BLUE/GREEN deployment strategy for database workloads. - Maintains a dedicated fallback node pool so database pods can continue to be scheduled when their primary node pool is unavailable or resource-constrained. - Prevents infrastructure reconciliation from reverting operational changes while a related remediation is being deployed. - Reduces the likelihood that log-backlog processing can compete with critical database workloads for resources. We regret the disruption caused by these incidents. We are continuing to review our upgrade, capacity-management, and infrastructure-reconciliation procedures to reduce both the likelihood and impact of similar events.

3. investigating Jul 13, 2026, 02:12 PM UTC

   Root Cause Analysis — July 13, 2026 Incident summary On July 13, 2026, we experienced two service interruptions related to the same underlying issue: 07:37–08:36 UTC 13:52–14:12 UTC Root cause: The incident began when our Kubernetes cluster stopped forwarding logs due to an issue with Fluentd, our log collection service. To restore log forwarding, we initiated a non-disruptive upgrade of the Kubernetes node pools. Based on the expected behavior of this procedure, the upgrade was not anticipated to affect service availability. When the upgrade reached the node pool hosting ScyllaDB, Fluentd began processing a backlog of accumulated logs. This caused an unexpected increase in CPU and memory consumption on the affected nodes. As a result, the cluster temporarily lacked sufficient resources to schedule the ScyllaDB pods, leading to service unavailability. Once the node-pool upgrade had started, the operation needed to complete before we could safely apply further changes, this extended the recovery time by approximately 20 minutes. During recovery, we developed a remediation plan to increase the resources available to the ScyllaDB workloads. While deploying that change, our Terraform controller reconciled the environment to its previous configuration and reverted part of the Fluentd update before the remediation was fully applied. This caused the second service interruption. Resolution and preventive measures: We have implemented a permanent remediation that: - Provides additional CPU and memory headroom for all database workloads. - Implementing BLUE/GREEN deployment strategy for database workloads. - Maintains a dedicated fallback node pool so database pods can continue to be scheduled when their primary node pool is unavailable or resource-constrained. - Prevents infrastructure reconciliation from reverting operational changes while a related remediation is being deployed. - Reduces the likelihood that log-backlog processing can compete with critical database workloads for resources. We regret the disruption caused by these incidents. We are continuing to review our upgrade, capacity-management, and infrastructure-reconciliation procedures to reduce both the likelihood and impact of similar events.

Read the full incident report →

Started Jul 13, 2026, 07:37 AM UTC· resolved Jul 13, 2026, 08:36 AM UTC

1. investigating Jul 13, 2026, 07:37 AM UTC

   We are currently investigating an issue affecting the Visually platform. Following a scheduled Google Cloud (GCP) node upgrade, one of our databases failed to start correctly. As a result, the Allocator API is currently unavailable, and experiences are not being served by Visually. Our engineering team is actively working to identify the root cause and restore service as quickly as possible. We will provide another update as soon as we have more information.

2. resolved Jul 13, 2026, 08:36 AM UTC

   We have identified the issue, and the affected cluster is back online. Our team is currently performing additional validation checks to ensure all services are operating as expected. We will continue investigating the root cause of the incident and will provide further updates as more information becomes available.

Read the full incident report →

Started May 12, 2026, 10:38 PM UTC· resolved May 12, 2026, 11:19 PM UTC

1. investigating May 12, 2026, 10:38 PM UTC

   Allocator API went down.

2. resolved May 12, 2026, 11:19 PM UTC

   Allocator API recovered. We experienced a temporary service disruption caused by a Google Cloud Platform (GCP) infrastructure issue affecting instance availability in the region. While our failover infrastructure activated as designed, the underlying provider-level capacity issue also impacted secondary recovery resources. Services have been fully restored, and we have implemented additional resilience measures, including expanding to a third infrastructure pool to further strengthen redundancy and recovery capabilities. We apologize for the inconvenience and appreciate your patience.

Read the full incident report →

Started May 03, 2026, 07:15 PM UTC· resolved May 03, 2026, 07:18 PM UTC

1. investigating May 03, 2026, 07:15 PM UTC

   Recommendations API went down.

2. resolved May 03, 2026, 07:18 PM UTC

   Recommendations API recovered.

Read the full incident report →

Started Apr 27, 2026, 11:52 AM UTC· resolved Apr 27, 2026, 12:52 PM UTC

1. investigating Apr 27, 2026, 11:52 AM UTC

   The Editor's quick preview (where you see the preview of the website) is down due to Outage of our 3rd party proxy service at Bright Data. We are working to mitigate this issue ASAP

2. investigating Apr 27, 2026, 12:12 PM UTC

   We’ve mitigated the issue by switching to an alternative proxy network. While this network is slightly slower, it should remain stable and functional until Bright Data resolves the underlying problem. Thank you for your patience.

3. resolved Apr 27, 2026, 12:52 PM UTC

   Bright data has resolved the issue - back to normal

Read the full incident report →

Started Apr 23, 2026, 11:09 AM UTC· resolved Apr 23, 2026, 11:13 AM UTC

1. investigating Apr 23, 2026, 11:09 AM UTC

   Recommendations API went down.

2. resolved Apr 23, 2026, 11:13 AM UTC

   Recommendations API recovered.

Read the full incident report →

Started Apr 16, 2026, 03:08 PM UTC· resolved Apr 16, 2026, 03:08 PM UTC

1. resolved Apr 16, 2026, 03:08 PM UTC

   We are currently experiencing service degradation while ingesting new data events into our Snowflake data warehouse. (no data loss) Snowflake has reported an incident affecting our cluster. As a result, our platform may experience delays in report data updates. We are monitoring the situation closely and will share updates as they become available. https://status.snowflake.com/incidents/0lj68wp0lphb

Read the full incident report →

Started Mar 09, 2026, 12:19 PM UTC· resolved Mar 09, 2026, 01:07 PM UTC

1. investigating Mar 09, 2026, 12:19 PM UTC

   We are currently experiencing intermittent issues affecting our Editor and Dashboard due to a Google Cloud outage. Our team has escalated the issue to Google Cloud and is awaiting further updates. We will share more information as it becomes available.

2. investigating Mar 09, 2026, 12:40 PM UTC

   Update: The Dashboard and Editor are back to normal. Google Cloud hasn’t resolved the underlying issue yet, so sporadic errors may still occur. We’ll keep monitoring and update if anything changes.

3. resolved Mar 09, 2026, 01:07 PM UTC

   Google Cloud has resolved the issue, and all services are now working as expected.

Read the full incident report →

Started Jan 08, 2026, 10:19 PM UTC· resolved Jan 08, 2026, 11:02 PM UTC

1. investigating Jan 08, 2026, 10:19 PM UTC

   We’re currently investigating an issue caused by a recent Shopify change that appears to have broken backward compatibility in certain Checkout UI Extension components. As a result, some elements that were expected to be hidden are now being rendered (including raw JSON content). As a precaution, we’ve temporarily removed all checkout extension components across all stores. No manual action is required on your end, and there’s no need to pause experiences individually. We’re actively working on a fix and will share an update shortly.

2. investigating Jan 08, 2026, 10:46 PM UTC

   We’ve resolved the issue impacting live experiences. Widgets are now rendering correctly and displaying as expected. If you previously paused an experience or removed mount points, it’s now safe to resume or restore them. As a result, all currently live experiences should be functioning as expected, and the live issue is considered resolved. Current limitation At this time, editing existing checkout experiences or creating new checkout experiences may not work as expected. We’re actively working on this and will share another update as soon as this is addressed. Thanks for your patience, and please let us know if you notice anything unexpected on live traffic.

3. resolved Jan 08, 2026, 11:02 PM UTC

   The issue caused by a recent Shopify change affecting Checkout UI Extension components has been resolved. The affected elements are now rendering correctly, and unintended content (including raw JSON) is no longer displayed. All checkout extension components have been safely restored across stores. No action is required on your end. We’ll continue to monitor the situation closely and will share updates if anything changes. Thank you for your patience.

Read the full incident report →

Started Oct 20, 2025, 08:45 AM UTC· resolved Oct 21, 2025, 04:07 AM UTC

1. investigating Oct 20, 2025, 08:45 AM UTC

   We are currently experiencing issues due to an officially reported AWS outage, which is impacting some third-party services we rely on. As a result, our new AI Component Generator is temporarily unavailable — new chats cannot be initiated at this time. AWS is actively investigating the issue, and we will continue to monitor the situation closely. Updates will be posted here as soon as more information becomes available.

2. resolved Oct 21, 2025, 04:07 AM UTC

   Issue with the AI provider resolved, AWS officially confirmed that the issue was resolved

Read the full incident report →

Started Jul 30, 2025, 07:32 AM UTC· resolved Jul 30, 2025, 07:32 AM UTC

1. resolved Jul 30, 2025, 07:32 AM UTC

   Our dashboard is currently partial unavailable and may display incorrect error messages. We are actively working to resolve the issue and will provide updates as soon as possible. Thank you for your patience.

Read the full incident report →

Started Jun 24, 2025, 10:08 AM UTC· resolved Jun 24, 2025, 10:18 AM UTC

1. investigating Jun 24, 2025, 10:08 AM UTC

   Allocator API went down.

2. resolved Jun 24, 2025, 10:18 AM UTC

   Allocator API recovered.

Read the full incident report →

Started Apr 08, 2025, 12:09 AM UTC· resolved Apr 08, 2025, 01:55 AM UTC

1. investigating Apr 08, 2025, 12:09 AM UTC

   Recommendations API went down.

2. investigating Apr 08, 2025, 12:38 AM UTC

   Recommendations API recovered.

3. investigating Apr 08, 2025, 01:38 AM UTC

   Allocator API went down.

4. investigating Apr 08, 2025, 01:39 AM UTC

   Recommendations API went down.

5. investigating Apr 08, 2025, 01:54 AM UTC

   Recommendations API recovered.

6. resolved Apr 08, 2025, 01:55 AM UTC

   Allocator API recovered.

Read the full incident report →

Started Apr 07, 2025, 11:35 PM UTC· resolved Apr 08, 2025, 01:58 AM UTC

1. investigating Apr 07, 2025, 11:35 PM UTC

   We're currently experiencing a partial outage due to issues with Google Cloud Platform (GCP), which is affecting the delivery of user experiences. Our team is actively working on mitigation and monitoring the situation closely.

2. resolved Apr 08, 2025, 01:58 AM UTC

   The issue with Google Cloud Platform (GCP) has been resolved. All services are now fully operational, and user experiences are being served as expected. We’re continuing to monitor for stability. Thank you for your patience.

Read the full incident report →

Started Feb 25, 2025, 07:01 PM UTC· resolved Feb 25, 2025, 07:07 PM UTC

1. investigating Feb 25, 2025, 07:01 PM UTC

   Allocator API went down.

2. investigating Feb 25, 2025, 07:01 PM UTC

   Recommendations API went down.

3. resolved Feb 25, 2025, 07:07 PM UTC

   Allocator API and Recommendations API recovered.

Read the full incident report →