Machine-learning models are commonly described by their number of trainable parameters, but that figure can be misleading. A model may contain many parameters that do not create genuinely independent ways of influencing its output, while a compact model may use nearly every parameter in a distinct and meaningful way. Effective dimension is a way of quantifying that difference.
Consider two models with parameters θ1 and θ2. In the first, the parameters affect the output independently, so the model can vary in two independent directions. In the second, the output depends only on their sum, so many parameter pairs produce exactly the same function. Although the model formally has two parameters, it has only one degree of freedom. This illustrates the central idea: parameter count does not necessarily represent functional complexity.
Effective dimension is a scale-dependent measure of information complexity, originally developed in statistical information theory to count the truly active parameters in a model based on the size of available data. It has since been adopted in classical and quantum machine learning to evaluate a model's expressivity, trainability, and ability to generalize. Unlike accuracy, loss, or raw parameter count, it asks how many distinguishable parameter directions a model has across a chosen parameter region and data distribution.
Importantly, effective dimension is related to generalization. Under the framework's assumptions, it can provide a complexity-dependent bound on generalization performance. This is the first part of a two-part series; the second part will show how to implement effective dimension calculations in machine-learning experiments.
Source: Towards Data Science · Summarized by HeadlinesBriefing