Prismatic incident

Degraded Database Performance in us-east-2 Region

Major Resolved View vendor source →

Prismatic experienced a major incident on January 23, 2023 affecting GraphQL API and Web App and 1 more component, lasting 16h 16m. The incident has been resolved; the full update timeline is below.

Started
Jan 23, 2023, 10:10 PM UTC
Resolved
Jan 24, 2023, 02:26 PM UTC
Duration
16h 16m
Detected by Pingoru
Jan 23, 2023, 10:10 PM UTC

Affected components

GraphQL APIWeb AppIntegration RunnerOAuth 2.0 Refresh

Update timeline

  1. investigating Jan 23, 2023, 10:10 PM UTC

    We are currently investigating an issue where database performance is noticeably degraded across all services in the us-east-2 region. A support issue has been raised with AWS, as logs and metrics seem to indicate a potential problem with the hosted service.

  2. monitoring Jan 23, 2023, 10:53 PM UTC

    The observed issue seems to be transient in nature. We are working with AWS support to understand and identify what appears to be a potential issue with the underlying database hosting service, perhaps related specifically to networking. All services have returned to operational status with normal performance metrics. We will continue to monitor the situation.

  3. resolved Jan 24, 2023, 02:26 PM UTC

    We monitored the issue through the evening and this morning and things appear to be working normally. We are still working with AWS support to try and identify the underlying cause. If the issue returns we will open a new incident.