Machine learning optimization evolved from deterministic gradient descent to stochastic methods when practitioners faced computational limitations with large datasets. Traditional gradient descent calculates the exact gradient using all training data, creating bottlenecks as datasets grew beyond computational capabilities. This mathematical approach remains fundamental for understanding optimization processes in machine learning.

The mathematical foundation lies in minimizing loss functions through partial differentiation, as shown in the slope and intercept formulas derived from the Mean Squared Error. This approach works perfectly for small datasets but struggles with scalability. Deriving these formulas involves setting partial derivatives equal to zero and solving for the parameters that minimize error.

Stochastic gradient descent emerged as a practical solution by using random subsets of data for each update step. This method trades exact convergence for computational efficiency, allowing models to train on massive datasets that would otherwise be impossible to process with traditional optimization techniques. Modern deep learning relies heavily on this approach.