Simon Data Outage History

Services appear to be back to healthy. We're investigating the root cause.

Following some difficulties rolling over to a new Zendesk contract, we are now able to submit and edit tickets in Support Center. We are actively monitoring the issue for any other outstanding failure points.

We are tracking a service event at AWS in us-east-1 that is impacting Amazon Redshift availability. Some customers may see delayed or failing data refreshes. AWS has resolved the incident and we are working towards recovery.

AWS has resolved the upstream Redshift issue, and we are observing recovery across affected workloads. Queue depth is decreasing and queries are completing, though some processing may remain slower than normal as the backlog clears. A subset of customers may have seen delays in data refreshes, syncs, journeys, and parts of the Simon Web App during the impact window; some customers were not affected. We are continuing to monitor for full stability and will post another update once recovery is complete. AWS status: https://health.aws.com/health/status

Data pipelines experienced an unexpected delay due to an AWS access issue. This has since been resolved and we are actively ensuring pipelines finish as quickly as possible.

A fix has been implemented and we are monitoring.

Customers utilizing Snowflake services in the following regions are experiencing service disruption: Regions Impacted: AWS - US West (Oregon) AWS - Europe (Ireland) AWS - Asia Pacific (Mumbai) Users are unable to execute queries against Snowflake databases, tables, and views. Error message presented: "SQL execution internal error" or similar messages. Snowflake has identified the root cause and is actively developing a fix -- https://status.snowflake.com/

Snowflake continues to make progress here, there are some aspects of data sharing that are now operational.

Journeys service has been restored and is now operational. We are actively monitoring the service to ensure full recovery.

A fix has been implemented and we are monitoring the results.

All services are up and back to operational. AWS has resolved the last of the Redshift issues that have been impacting us. Our batch processing is continuing as scheduling and the team is monitoring progress.

AWS had an outage that impacted a number of services that we rely on, including our ability to update our Status Page in a timely manner. The only impacted region was us-east-1. Journeys, Syncs, and Pipes have delayed processing that we are actively recovering from. Incoming events were delayed. More information on the outage. https://health.aws.amazon.com/health/status

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results. Web App may be slower to load.

Access to the Simon Documents pages have been restored and we are monitoring the results. We apologize for the downtime.

Sparkpost indicated a fix has been implemented and Simon is monitoring the results as we recover flows and journeys.

We are still monitoring as we recover any failed flows or journeys.

We have a few confirmed reports that the Web App is available for some users, but performance may be degraded. We continue to monitor the situation and will provide updates here.

At 03:40 PM PDT on July 30th AWS noticed increased error rates and latencies for some service APIs within the US-EAST-1 region. At 03:59 PM PDT on July 30th AWS determined it was due to the Kinesis service in US-EAST-1. The root cause was identified and they began working on a solution As of July 30th at 09:55 PM PDT AWS fully recovered Kinesis errors and closed the issue Simon Data has been recovering failed journeys since the service has returned to normal. We are actively working through the backlog and will provide another update once we are recovered.