Prodly experienced a minor incident on April 12, 2023 affecting AppOps Deployment and AppOps, lasting 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 12, 2023, 06:13 PM UTC
An interruption in the Prodly AppOps service has occurred for a subset of accounts. Users may experience timeouts when attempting to open data sets. The issue has been identified and a fix is being formulated.
- monitoring Apr 12, 2023, 06:42 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 12, 2023, 06:47 PM UTC
This incident has been resolved.
- postmortem Apr 19, 2023, 03:38 AM UTC
# April 12th Incident RCA ## Summary * On April 12th, starting approximately 14:00 UTC, a production incident occurred where customers were receiving “Read timed out” or “Exceeded maximum time allotted for callout” errors within the Data Set Editor.. * The error would appear on Data Sets using a Source Org Schema, or when changing the Source Org Schema. ## Team Response * The team was made aware of the outage after several users reported errors using the Data Set Editor. * The team alerted the on-call DevOps team who actively troubleshooted the issue regarding specific API calls timing out. * The root cause was discovered to be a change to Salesforce’s list of public IP addresses which triggered an update to an internal whitelist of IP addresses allowed to use a specific Prodly API that queries Salesforce for and returns an org’s schema. * The internal whitelist was intended to block requests not coming from Salesforce. * The automatic update overwrote a manual change which added an IP address of a new load balancer that was recently added to Prodly’s infrastructure, causing traffic originating from this load balancer to be blocked for a subset of customers. * Filtering of traffic based on the IP whitelist was disabled, which allowed all traffic through to the API, which resolved the issue at approximately 18:30 UTC. ## Mitigation Plan * This security measure was associated with our AWS infrastructure built on Elastic Beanstalk which will be retired with our 12.8 release. * We are re-evaluating the requirements of IP whitelisting for our new AWS infrastructure built on Kubernetes. * Any IP enforcement in the future will be updated to include the IP address of Prodly’s internal load balancer, so that future updates will not impact customers.