Researchers discovered that diverse language models, including geometrically separable features, emerge in number representations across architectures. This phenomenon occurs when models trained on natural text develop periodic Fourier features at intervals like T=2, 5, or 10. While all models capture these periodic patterns, only some achieve mod-T geometric separability—the ability to linearly classify numbers based on their modulus. The study shows this capability depends on specific training conditions, not just architecture.

The research identifies key factors enabling geometric separability: data composition, model design choices, optimizer behavior, and tokenization methods. Notably, mod-T separability requires sparse Fourier domain features but demands additional structural alignment. Empirical analysis reveals two distinct pathways: models can acquire these features through general language data containing number co-occurrences or via explicit multi-token arithmetic problems. For example, interactions between numbers in sentences or multi-step addition tasks trigger the development of separable representations.

This work highlights convergent evolution in AI—unrelated models independently evolve similar features from disparate signals. The findings matter for understanding how numerical reasoning emerges in language models. By isolating conditions that produce mod-T separability, researchers gain tools to engineer better number-handling capabilities. The study’s technical depth offers practical insights for improving arithmetic performance in AI systems without specialized math training.