Neural networks excel at complex tasks like image classification and language modeling, but regression suffers from 'spectral bias' - a tendency to learn low-frequency patterns before high-frequency ones. This phenomenon, documented since 2019, makes high-frequency function fitting notoriously slow and requires extensive training epochs.

The Neural Tangent Kernel framework explains spectral bias through linear dynamical systems, showing convergence rates inversely proportional to frequency. Other research links this bias to activation function spectra, particularly for ReLU and hyperbolic tangent functions. Remedies include second-order optimization, SIREN networks with periodic activations, and Fourier features that embed inputs in higher-dimensional spaces.

Authors Conor Rowan and Finn Murphy-Blanchard propose 'sequential fitting' as an alternative perspective. Their experiments with two-hidden-layer MLPs using tanh activations show networks fit target functions starting from domain boundaries, building oscillations incrementally. For v(x) = sin(26πx), training used ADAM optimization with learning rate 5×10^-3.

This boundary-driven learning explains why high-frequency functions challenge neural networks, particularly in scientific computing where multi-scale problems require oscillatory solutions. Their analysis of the learned basis functions reveals step-function-like structures that illuminate the underlying mechanism. The work suggests spectral bias stems from geometric rather than purely spectral constraints.