Increased error rates in Execution Pipeline (US Commercial)
Affected: Integration Runner
Prismatic Outage History
Prismatic is up right nowPrismatic had 9 outages in the last 2 years totaling 9h 43m of downtime — averaging 0.4 incidents per month.
There were 9 Prismatic outages since October 7, 2024 totaling 9h 43m of downtime. Each is summarised below — incident details, duration, and resolution information.
Partial Service Outage in EU-WEST-1 Region
Timeline · 1 update
- resolved Sep 03, 2025, 06:37 PM UTC
A routine update caused a service interruption of the execution pipeline for some customers for approximately 5 minutes. A rollback was performed and service was restored. The Prismatic team is investigating the root cause and will ensure a subsequent update does not contain the same regression.
Resolved: Web App and Embedded UI Outage (GovCloud Only)
Timeline · 1 update
- resolved Jun 15, 2025, 01:32 PM UTC
We experienced an outage impacting the Web App and Embedded UI in the GovCloud environment. The incident began at 9:19 a.m. CT on June 14, 2025 and was fully resolved at 8:02 a.m. CT on June 15, 2025. During the outage, users were unable to access the web app or embedded screens. Integration executions continued to run as expected and API access was not impacted. Commercial Cloud environments were not impacted.
Execution pipeline service degradation on EU-WEST-2
Timeline · 1 update
- resolved Jun 05, 2025, 07:23 PM UTC
At approximately 7:00AM Arizona the Prismatic team received several alert notifications for failed executions. Upon investigation it was discovered that a very large volume of executions had started, and autoscaling was not configured with a high enough upper threshold to accommodate. The Prismatic team took action to raise the thresholds and align them with other high-volume stacks, and service was fully restored by 7:20AM Arizona. We will audit and address any similar situations we discover on other high-volume stacks.
Increased error rates in Execution Pipeline
Affected: Integration Runner
Timeline · 5 updates
- investigating May 15, 2025, 09:10 PM UTC
At approximately 1:47 PST the Prismatic team received an alert for an increase in error rates in the execution pipeline. We have started a rollback procedure that we expect will resolve the issue, and then we will address the underlying root cause.
- investigating May 15, 2025, 09:34 PM UTC
We have completed rollback of all stacks the execution pipeline is back to fully operational status. We will continue our investigation to determine root cause and will keep this incident updated as we progress.
- investigating May 15, 2025, 09:34 PM UTC
We are continuing to investigate this issue.
- investigating May 15, 2025, 09:45 PM UTC
We have identified the root issue and are working on a fix.
- resolved May 15, 2025, 09:52 PM UTC
This incident has been resolved.
Embedded Service Interruption
Affected: Embedded
Timeline · 3 updates
- identified Apr 23, 2025, 05:41 PM UTC
A regression was introduced in the Embedded system that caused a Major disruption in service, specifically for organizations that use the i18n features. The issue was identified and a fix has been deployed. Prismatic is now confirming with impacted customers that the issue is resolved.
- monitoring Apr 23, 2025, 05:42 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 23, 2025, 05:43 PM UTC
Confirmed back to fully operational status.
Prismatic Web Application Hard Crashing
Timeline · 1 update
- resolved Dec 17, 2024, 03:56 PM UTC
At approximately 8:57am CT an update to the Prismatic web application was deployed that contained updated dependencies. Unfortunately there was a misconfiguration that was not problematic in development and CI environments, but caused crashes in the production environments. The Prismatic engineering team received alerts immediately upon release that there were problems and immediately started rolling back to the previous version. Full functionality was restored by 9:30am CT. The Prismatic engineering team is investigating root cause and will properly address this issue and further investigate how to improve the test coverage so as to minimize the likelihood and impact of similar situations in the future.
Partial Outage for US-EAST-2 Region
Affected: GraphQL APIWeb AppIntegration RunnerOAuth 2.0 Refresh
Timeline · 3 updates
- investigating Nov 05, 2024, 09:38 PM UTC
We are currently investigating an issue affecting the production database in the US-EAST-2 region. Performance has been degraded and has affected some executions as well as the ability to use the embedded marketplace and related services. The Prismatic team is actively investigating the issue and will provide continuous updates as we work towards resolution.
- monitoring Nov 05, 2024, 09:51 PM UTC
We have identified and mitigated the issue affecting the database in the US-EAST-2 region. All metrics have returned to normal operating levels. We will continue to monitor the situation.
- resolved Nov 05, 2024, 10:38 PM UTC
This incident has been resolved.
Degraded services in US-EAST-2 Regions
Affected: Integration RunnerOAuth 2.0 Refresh
Timeline · 6 updates
- investigating Oct 07, 2024, 07:39 PM UTC
The Prismatic team started seeing increased error rates in multiple services starting at 13:27 Mountain. The errors appear to originate in the underlying AWS services. We have reached out to our contacts at AWS for more information and to bring increased awareness to this issue. We will continue to monitor the situation and work towards resolution.
- investigating Oct 07, 2024, 07:40 PM UTC
We are continuing to investigate this issue.
- identified Oct 07, 2024, 07:44 PM UTC
We continue to monitor the situation.
- identified Oct 07, 2024, 07:50 PM UTC
AWS has confirmed an issue with S3 and related services in their us-east-2 region. They are working towards resolution. We will continue to monitor the situation and provide updates as they are available.
- monitoring Oct 07, 2024, 07:59 PM UTC
It appears that the relevant issues have been identified and addressed by the AWS teams. We will continue to monitor for new additional problems and provide updates until we are confident that operation has been completely restored.
- resolved Oct 07, 2024, 08:08 PM UTC
It appears the issue has been completely resolved and AWS is working to understand the root issue(s). We will open a new issue on this status page if any further problems emerge.