HeadlinesBriefing favicon HeadlinesBriefing.com

PSSA: Rust AI Model Beats Transformers

Hacker News •
×

PSSA (plastic state-space architecture) is a small language model written in Rust from scratch, without PyTorch or TensorFlow. Unlike transformers that score every token pair, PSSA processes text one token at a time through a recurrent state-space layer with an episodic memory bank. On 12.7M tokens of Wiki Text-103, PSSA reached 3.98 training cross-entropy vs the transformer's 4.43—a 0.45 nat gap. On unseen data, PSSA scored 3.997 vs 4.429, with 24.1% next-token accuracy vs 18.0%. Generating 200 tokens took 226ms for PSSA vs 2,735ms for the transformer—12x faster on CPU. Key features include learned continuous state matrices, a 512-slot hyperbolic memory bank, plastic weights with refractory gating, and closed-form consolidation via ridge regression. These are 1.5M-parameter research prototypes, not production models. Text quality remains poor for both, but PSSA demonstrates superior learning efficiency and speed.

The architecture differs fundamentally from transformers: cost grows linearly with sequence length instead of quadratically, and context isn't re-read at every step. A recurrent model carries fixed-size state forward, while transformers re-read their entire context window per token.

PSSA uses hand-written linear algebra in Rust with a CUDA training path and scalar CPU reference implementation (max gradient difference 2.98e-8). The comparison focuses on learning efficiency, not fluency.