OneSchema experienced a critical incident on June 14, 2023 affecting Dashboard and Embeddable products and 1 more component, lasting 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 14, 2023, 06:13 PM UTC
We are currently investigating this issue.
- identified Jun 14, 2023, 06:22 PM UTC
The issue has been identified and a fix is rolling out.
- resolved Jun 14, 2023, 06:24 PM UTC
Roll out complete -- Systems functioning
- postmortem Jun 15, 2023, 08:01 PM UTC
# Impact * Starting at 11:06 AM, OneSchema experienced an outage across US, EU, and CA regions after an update to our internal queuing service caused the web service `app.oneschema.co` to go down in those regions. A build that had an error only got partially caught by CI, leading to some regions being impacted. At 11:21 AM, the rollback to the queuing service was deployed and the incident was resolved. Users opening embeds in those regions did not open during the outage and return 502 responses and users visiting the admin dashboard during the outage would also see a 502 error page. # Path forward * We have prioritized updates to our CI system to prevent any builds with end to end test failures from reaching any of our production regions. * We will be investigating making our CI system atomic, immediately rolling back all regions if one region experiences a failure as part of the deploy process. We apologize for the downtime caused by this outage. Please reach out to your dedicated OneSchema Support or your dedicated Account Manager if you have any specific questions about the impact of this outage on your system.