A new mathematical framework reveals that training examples act as vector fields in neural network parameter space, and their Lie brackets can quantify how order-dependent gradient updates really are. While Bayesian learning treats training data as unordered, gradient descent exhibits significant order sensitivity that can be measured and analyzed. This work builds on Dherin's 2023 research connecting Lie brackets to implicit biases in neural network training.

The researchers computed Lie brackets between training examples using a MXResNet architecture trained on the CelebA dataset for 5000 steps with Adam optimizer. By examining parameter checkpoints throughout training, they discovered that Lie bracket magnitudes correlate tightly with gradient magnitudes across different parameter tensors. This suggests that non-commutativity effects depend primarily on the vector field structure of individual examples and training progress, rather than varying significantly across different parts of the network.

Interestingly, the analysis revealed that features like Black_Hair and Brown_Hair predictions showed high sensitivity to training order, potentially flagging modeling issues. Since these features are mutually exclusive in the dataset but the model predicts them independently, the loss function's assumption of independence may create inconsistencies. The framework provides a new tool for understanding and potentially debugging order-dependent behaviors in neural network training.