Many deep learning engineers default to using Adam W with its standard parameters, often setting the learning rate to 3e-4. This common practice, influenced by community norms and even AI suggestions, overlooks crucial aspects of Adam's optimization dynamics.

The Adam optimizer, designed to adapt step sizes per parameter using first and second moment estimates of gradients, can fail spectacularly, especially in non-stationary or challenging optimization landscapes. The default decay parameters, β₁=0.9 and β₂=0.999, can lead to issues where the optimizer converges to suboptimal solutions or fails to converge altogether, as demonstrated in research by Reddi, Kale, and Kumar.

Understanding how Adam works, particularly the role of these decay parameters and their bias correction, is vital. Modifying β₂ from 0.999 to 0.95 and β₁ from 0.9 to 0.5, as one engineer discovered, can be the key to successful training on difficult problems, rather than simply relying on default settings.