Researchers revisit the necessity of self-attention in sequence modeling, proposing an alternative architecture based on Grassmann flows. Instead of computing an L by L attention matrix, their Causal Grassmann layer linearly reduces token states, encodes local pairs as two-dimensional subspaces via Plucker coordinates, and fuses these geometric features back into hidden states through gated mixing. This approach operates on a finite-dimensional manifold rather than an unstructured tensor space.
On the Wikitext-2 benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of Distil BERT slightly outperforms a Transformer head, achieving best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. The authors analyze the complexity of Grassmann mixing, showing linear scaling in sequence length for fixed rank, and argue that manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.
Source: Hacker News · Summarized by HeadlinesBriefing