Decision Trees classify data by creating nested decision rules that partition feature space into pure regions. But how do they choose where to split? The answer lies in entropy, a measure of information that quantifies how mixed or pure a dataset is. Pure nodes have zero entropy while impure ones have higher values.

To train a Decision Tree, the ID3 algorithm uses entropy to calculate information gain - the reduction in uncertainty achieved by each potential split. For each feature and cutoff value, the algorithm computes the difference between pre-split entropy and the weighted average entropy of child nodes. The split yielding maximum information gain becomes the decision node.

While entropy is the standard metric, Gini impurity offers an alternative that's faster to compute since it avoids logarithms. Both methods produce comparable trees, though entropy may be more robust for imbalanced datasets. Decision Trees offer simplicity and interpretability but suffer from instability - small data changes can drastically alter the tree structure. They also risk overfitting if allowed to grow too deep without proper stopping criteria like minimum samples per leaf or maximum depth.