Prodly experienced a major incident on May 31, 2023 affecting AppOps Deployment, lasting 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 31, 2023, 08:36 PM UTC
An interruption in the Prodly AppOps service has occurred. Users may encounter an error claiming that an active license cannot be found up on deployment attempts. A fix is being implemented, and we will continue providing further updates.
- resolved May 31, 2023, 08:49 PM UTC
This incident has been resolved.
- postmortem Jun 01, 2023, 10:38 PM UTC
## Summary * **Incident #1:** On May 31st, starting around 12:30 UTC, we received multiple reports of customers not being able to load the Data Set Editor for certain Data Sets. * The issue was isolated to Data Sets where a Schema Org was selected that was a Salesforce Summer ‘23 preview sandbox. * **Incident #2:** On May 31st, during the rollout of the 12.11 release of AppOps, our QA team discovered an issue deploying data between orgs that was also reported by several customers around the time of the release. * Symptoms included receiving a “No associated active license” error message after invoking a data deployment. ## Team Response * **Incident #1:** The engineering team isolated the issue loading Data Sets to a bug introduced in the Salesforce Summer ‘23 release REST API. Data Sets using a Schema Org sandbox that was not upgraded to Summer ‘23 did not experience the issue. * The engineering team reported the bug to Salesforce and also implemented a workaround in AppOps to allow these Data Sets to be opened successfully. * The fix was included in the 12.11 release which was deployed on May 31st at 20:20 UTC. * **Incident #2:** The engineering team identified the issue to be a misconfiguration of one of the application servers created for the 12.11 release. This misconfiguration was causing an internal license check to fail, which would automatically abort all submitted deployments. * A fix was deployed at 20:40 UTC which resolved the issue. ## Mitigation Plan * **Incident #1:** The QA team will ensure validation testing on new Salesforce releases happens sooner, before sandboxes are automatically upgraded. The engineering team will be following up with Salesforce Support to ensure the bug is acknowledged and fixed. Meanwhile, the workaround will remain in place. * **Incident #2:** The DevOps team is reviewing the automation scripts related to release deployments to ensure the bug resulting in server misconfiguration is fixed and that there are no other issues.