The theory behind stacking in machine learning holds that it works best when base models make different kinds of errors. A meta-model can learn to trust each model where it is reliable, but if all base models make the same mistakes, there is little complementary information to combine. This makes stacking's benefit a testable claim: it should scale with how differently the base models err.
To test this, the author built two conditions on a synthetic binary classification dataset of 2,000 samples and 20 features. The first used diverse base models: logistic regression, a decision tree, an SVM, and k-nearest-neighbors, each drawing decision boundaries in a fundamentally different way. The second used correlated models: random forest, gradient boosting, and extra-trees, three variants of tree ensembles. Results came from repeated stratified 5-fold cross-validation with 3 repeats, rather than a single train/test split.
In the diverse condition, stacking reached 0.830 versus 0.820 for the best single model, a meaningful gap that matches the theory. In the correlated condition, stacking's edge nearly vanished, at 0.835 versus 0.834, a difference well within the error bars. Voting, which simply averages models, underperformed the best single model in both conditions, because it cannot down-weight weaker models. Stacking's meta-model can learn how much to trust each base model, which is why it avoided this problem.
Source: Towards Data Science · Summarized by HeadlinesBriefing