GitHub user samir-sarwar released samir-vllm, a small LLM inference engine built from scratch in C++ and CUDA. The project includes 6+ hours of YouTube tutorials and 85+ pages of handwritten notes. It teaches LLM inference from first principles, covering every step from loading model weights to generating tokens on GPU.
The series explains the transformer pipeline, including embedding, RMSNorm, RoPE, Q/K/V projections, attention, and feed-forward networks. It features a fixed Llama 3.2 1B weight layout, a local BPE tokenizer, and CUDA kernels for embedding, normalization, and attention. The engine supports paged and continuous batching, static scheduling, and greedy response generation with Instruct chat formatting.
The codebase uses CMake, ICU for tokenization, and BF16 precision. A companion PDF offers 80+ pages of notes and diagrams. Current scope is limited to the Llama 3.2 1B path with a 2,048-token context and a default 128 MiB KV cache adjustable via --kv-cache-mib.
Source: Hacker News · Summarized by HeadlinesBriefing