Two years ago, discussions about Kolmogorov-Arnold networks began dominating tech news feeds. These networks replace the scalar weights of a conventional neural network with learnable functions. The change sounds small, but it raises a striking possibility: instead of only asking whether a network makes accurate predictions, we can explore the functions it learns and perhaps turn a trained network back into an equation.
To understand KANs, the author builds them from scratch and experiments with increasingly flexible edge functions in Lab Layer, an early concept tool for prototyping and visualising networks. The experiments start with simple quadratics, move to B-splines and then Chebyshev polynomials, and culminate in attempts to recover the Lorenz Equations from synthetic data.
Proposed in 2024, KANs are inspired by the Kolmogorov–Arnold representation theorem, which states that a continuous multivariate function on a bounded domain can be represented through compositions and sums of continuous single-variable functions. In a standard network, each edge contributes a scalar weight that is multiplied by its input before the node sums its inputs and applies an activation function. In a KAN, each edge instead applies a learned univariate transformation, and the node sums these transformed outputs. This moves the nonlinearity from the node to the individual edges and parameterises those edge functions using B-splines or other suitable families.
The author's early step was implementing training on edges, learning a parameterised function rather than a single value. Later sections test whether such networks can recover a physical system, using the Lorenz Equations as the benchmark, and distinguish between making accurate predictions and actually recovering the underlying equation.
Source: Towards Data Science · Summarized by HeadlinesBriefing