Gigatoken is a new language model tokenizer designed for extreme speed, achieving throughput of GB/s on various hardware. It claims to be up to ~1000x faster than existing solutions like Hugging Face's tokenizers and tiktoken, even though those already utilize multithreaded Rust.

Gigatoken offers two modes: a compatibility mode that acts as a drop-in replacement for Hugging Face Tokenizers or Tiktoken, and a native Gigatoken API for maximum performance. The native API allows Rust to handle data directly, minimizing overhead and maximizing parallelism. Benchmarks on an AMD EPYC 9565 processor show Gigatoken achieving 24.53 GB/s for GPT-2 tokenization, compared to 24.8 MB/s for Hugging Face and 36.0 MB/s for tiktoken.

This speed is attributed to heavy optimization of the pretokenization process using SIMD, minimized branching, and efficient caching of pretoken mappings. The project supports a wide range of CPUs and nearly all commonly used tokenizers, with detailed benchmarks available for numerous models including Llama, Qwen, and Gemma.