Bagging, or Bootstrap Aggregating, tackles model instability by training multiple models on different random data samples. Each model, like a juror with unique biases, votes on predictions. For classification, majority vote wins; for regression, predictions are averaged. This method reduces variance because individual errors cancel out, leading to more stable and reliable outcomes than any single model.

The technique's power comes from bootstrap sampling, where each training set is drawn with replacement. Roughly 63.2% of original data is unique to each sample, creating diversity. This independence is key; highly correlated models offer little improvement. Bagging excels with high-variance base models like deep decision trees, but it cannot fix a biased model—its bias remains in the ensemble.

Practical implementation is straightforward in libraries like scikit-learn. A crucial feature is out-of-bag (OOB) error estimation, providing a free validation score by using data left out of each bootstrap sample. While bagging itself is powerful, it's the foundation for Random Forests, which add random feature selection at each split to further decorrelate trees and boost performance.