Since Richard Sutton's influential 2019 essay on the bitter lesson, AI progress has leaned on general methods that scale computation and learn from raw data with minimal human priors. We argue that true intelligence learns end-to-end from messy inputs—finding patterns, building abstractions, and deriving meaning as efficiently as possible. As AI enters data-scarce domains like science, robotics, and biology, developing more general learning algorithms becomes critical.
Language models, AI's biggest success story, largely embrace the bitter lesson through scaling simple algorithms like the Transformer, next-token prediction, and test-time computation. Yet they still rely on hand-crafted tokenizers, limiting end-to-end learning. We believe byte-level models are foundational—not just for improving LLMs, but as a testbed for general learning methods.
In 2025, we developed hierarchical networks (H-Nets), addressing tokenization while moving toward architectures that learn from raw data across modalities. This first post explores scaling laws for H-Nets, showing compute efficiency gains up to 3-5x via learned tokenization. A flexible 2-stage H-Net outperforms a matched open-source model with 10x less compute and data, exhibiting novel behaviors like in-context tokenization.
Our results suggest H-Nets are a foundational architecture aligned with the bitter lesson, offering significant efficiency and scalability gains as we push AI to learn from all data.
Source: Hacker News · Summarized by HeadlinesBriefing