Flxpoint experienced a major incident on May 13, 2022 affecting API, lasting 3d 14h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 13, 2022, 12:53 PM UTC
We've identified an issue that is not allowing new Inventory Variants to be created. We are working on a resolution.
- identified May 13, 2022, 07:43 PM UTC
We have implemented and validated a fix for this issue in our test environments and we are actively working to roll this out to our production environments. This roll-out is estimated to take 10-24 hours for a complete resolution.
- identified May 14, 2022, 06:16 AM UTC
We are still actively rolling out this to our production environments. During this process, we detected that some "Get Inventory Primary" and/or "Get Inventory Secondary" jobs were timing out due to the increased load on our system. We have addressed this so that these jobs are now functioning as intended. We are striving to get this resolved as quickly as possible. Due to the nature of the incident, we expect this restriction on new inventory records to remain in place for ~6-12 hours.
- identified May 14, 2022, 05:41 PM UTC
Our team has continued to work through this overnight / today. This resolution is taking longer than initially expected but is making steady progress. During the roll-out, we detected degraded performance when retrieving Purchase Orders via our v1 API and Automation jobs. We also found some some failures related to Inventory Images as a result of this roll-out. We've addressed both of these bottlenecks so they are no longer experiencing degraded performance. Our team is still hard at work on this and we're striving to get this resolved as soon as possible.
- identified May 16, 2022, 11:55 AM UTC
Our team is still actively working through this and treating it as the highest priority item. There are only 2 resources remaining that are affected by this (before we can re-enable creating variants). We hope to have this completely resolved soon.
- monitoring May 17, 2022, 03:14 AM UTC
We have deployed a resolution for this incident and are seeing new inventory variants reflecting in our system as a result. We are monitoring the resolution
- resolved May 17, 2022, 03:39 AM UTC
We have verified this incident has been resolved. During the duration of this incident, we also detected that some of our integrators were getting intermittent 500s while POSTing /orders for our v1 API. This has been addressed with the resolution we've implemented. We apologize for the impact and duration of this incident and we will release a statement with more detail in the coming days.