Yello experienced a minor incident on November 18, 2021 affecting Yello Enterprise Production - US, lasting 6h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 18, 2021, 05:02 PM UTC
Between Wednesday 11/17 at 2pm CST and today 11/18 at 10:05 am CST, integration imports and exports were either running sporadically or not running at all due to an issue that we are still investigating. We have monitoring in place to alert us in these kinds of situations but it failed and we are still investigating to understand why. Integrations are running as expected now, but we are still working to understand why they stopped. We will follow up and provide additional information as we receive it.
- resolved Nov 18, 2021, 11:12 PM UTC
This incident is fully resolved and we have additional information to share about what happened. Our integrations job scheduler experienced an issue that prevented any job from being processed. All integrations stopped at 11 pm. We discovered the issue and resolved it at 10:05 am. We had monitoring in place for this scheduler, but this scenario was not covered. We have since updated our monitoring and added specific monitoring to cover this issue. We also added alerting if no integration jobs are processed.