Prismatic incident
Elevated error rate for integration runner invocations in US region
Prismatic experienced a major incident on June 26, 2023 affecting GraphQL API and Web App and 1 more component, lasting 5h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 26, 2023, 09:49 PM UTC
We are currently investigating this issue.
- investigating Jun 26, 2023, 10:03 PM UTC
We are continuing to investigate this issue.
- investigating Jun 26, 2023, 10:26 PM UTC
The engineering team is continuing to investigate issues related to the integration runner.
- investigating Jun 26, 2023, 10:46 PM UTC
An issue has been identified and our engineering team is implementing a fix.
- investigating Jun 26, 2023, 11:15 PM UTC
Our engineering team is continuing to implement a fix
- investigating Jun 26, 2023, 11:49 PM UTC
A fix has been put in place, and the integration runner service is returning to normal operation. Our engineering team is monitoring the service.
- monitoring Jun 27, 2023, 12:20 AM UTC
Our team is continuing to monitor integration runner performance.
- monitoring Jun 27, 2023, 12:44 AM UTC
Mitigation efforts continue on a failing piece of critical piece of infrastructure. API and application performance are negatively impacted and Integration Runner is operating at a lower capacity than normal. A fix has been implemented and rollout continues.
- monitoring Jun 27, 2023, 01:30 AM UTC
Rollout of the expected fix continues and the team continues to monitor the performance of the platform.
- monitoring Jun 27, 2023, 02:33 AM UTC
We are continuing to monitor for any further issues.
- monitoring Jun 27, 2023, 02:35 AM UTC
The team has completed rollout of the fix and things appear to be operating normally. We will continue to monitor the situation to ensure that no regressions appear.
- resolved Jun 27, 2023, 02:54 AM UTC
All indication are that performance across the platform has been restored. The incident has been resolved.