Next-Concept Modeling (NCM) introduces a two-stage, coarse-to-fine framework for text generation. A multiscale residual VQ-VAE encodes text into a hierarchy of discrete codes, while a transformer generates one code level at a time in parallel. This approach significantly improves efficiency compared to diffusion language models.
At matched parameter scales, NCM achieves better generative perplexity than the strongest baseline on LM1B while running 21 times faster. On Open Web Text, it remains competitive with the strongest diffusion model but operates 10 times faster. On the XSum summarization task, NCM matches the strongest baseline's ROUGE-L score while running 31 times faster.
Beyond standard generation, NCM demonstrates strong performance on reasoning tasks, specifically Sudoku, where it outperforms both autoregressive and diffusion baselines across all difficulty levels using fewer function evaluations. This work presents a compelling alternative for efficient parallel text generation.
Source: Hacker News · Summarized by HeadlinesBriefing