The LatentVLA architecture introduces a novel approach to autonomous driving by moving beyond natural language-based reasoning. Unlike traditional models that rely on language datasets, LatentVLA predicts discrete ego-actions directly from raw driving data using self-supervised learning. This eliminates the need for extensive labeled datasets and addresses the inefficiency of token-based representations. The system uses an encoder-decoder setup with latent actions, discretized via a Vector-Quantised VAE, to separate ego-centric and environmental dynamics. LatentVLA achieves state-of-the-art results on the NavSim benchmark, outperforming standard E2E architectures. Its integration with E2E models via knowledge distillation provides a compact reasoning backbone without sacrificing performance.

Training involves predicting latent actions from unlabeled data, forcing the model to learn predictive representations of driver behavior. The architecture uses a two-stage encoder-decoder system to disentangle ego-actions from environmental noise, ensuring accurate trajectory predictions. By quantizing continuous actions into a small codebook (only 16 tokens), LatentVLA simplifies the learning task while preserving pre-training knowledge. This approach allows the large Qwen2.5-VL model to distill into a much smaller 50M-parameter transformer, enabling efficient real-time deployment.

Evaluation on NavSim demonstrates LatentVLA's effectiveness, though the non-reactive simulation setup limits real-world applicability. The framework represents a significant step toward integrating VLM knowledge into traditional autonomous driving systems, offering a more efficient and scalable alternative to language-based methods.