Stacker experienced a major incident on April 7, 2022 affecting Stacker API and Airtable Schema Update and 1 more component, lasting 17h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 07, 2022, 10:34 PM UTC
We are currently experiencing a service degradation which is causing interruptions to onboarding, data syncs, and schema syncs for some apps. Our engineering team is actively investigating the issue. We will update this incident as we learn more and identify the cause and resolution.
- investigating Apr 08, 2022, 12:43 AM UTC
Our engineering team is still actively investigating this issue to find the root cause. At the moment, apps are expected to be able to do manual schema/data syncs (including new app onboardings). No apps will have automatic syncs at the moment. All other app functionalities are not affected.
- investigating Apr 08, 2022, 08:08 AM UTC
We are still looking into the issue with automatic data syncing. You can still use manual data and schema syncs, but automatic syncing will not work.
- identified Apr 08, 2022, 08:42 AM UTC
We have identified the root cause and are working on a fix. We will update as soon as the fix is live. In the meantime, please continue using manual data and schema syncs.
- identified Apr 08, 2022, 01:22 PM UTC
Thanks for your ongoing patience. We have a fix ready, and have started rolling it out as an update. The update is expected to take around 4 hours. During this time you may see intermittent issues such as: - Slower loading of the app - Record loading request timing out (the turtle icon) We will update the status page if the timeline changes, or if anything new happens.
- monitoring Apr 08, 2022, 03:33 PM UTC
The fix is now live, and data should now sync automatically. We’ll be monitoring the situation to make sure that we are in the clear.
- resolved Apr 08, 2022, 04:02 PM UTC
We are in the clear, the incident is resolved. Now that was intense—sorry for the hassle, and thanks for your incredible patience. The issue was caused by an overzealous security update that slowly took all of the server resources for itself. We will rest up, and then take steps to avoid this type of situation in the future.