Transformers, the architecture behind every major LLM since Google's 2017 "Attention Is All You Need" paper, are becoming a bottleneck. Justin Dangel, CEO of Subquadratic, calls them "one of the most important innovations in the history of computer science," but their dense attention mechanism requires massive computation—$50 billion in computing costs for Open AI alone this year, per Greg Brockman. As LLMs tackle reasoning and larger context windows, transformers struggle with scaling.

Startups are pursuing alternatives. Subquadratic claims its sparse attention model, Sub Q, rivals mainstream LLMs by dynamically selecting relevant words. Manifest AI replaces attention with "power retention," a rolling summary that drops less relevant data, demonstrating it on Star Coder and Qwen variants.

Liquid AI, an MIT spinout led by Ramin Hasani, pairs transformers with liquid neural networks inspired by worm brains. Its LFMs are 20% transformers, 80% liquid networks, run on $50 Raspberry Pis, and match rivals four times larger. With 34 million downloads, they're free for orgs under $10M revenue. These innovations could make LLMs faster, cheaper, and smarter.