The repo tiny‑vllm offers a compact, high‑performance LLM inference engine written in C++ and CUDA, positioned as a smaller counterpart to vLLM.

Developers receive full source code plus a step‑by‑step course that walks through loading a real model from Safetensors—specifically Llama 3.2 1B Instruct—and running the entire forward pass, from prefill to decode, with custom CUDA kernels.

The engine supports KV caching, static and continuous batching, and a Flash‑Attention‑like PagedAttention kernel. By teaching low‑level GPU math—bfloat16, RMSNorm, parallel reduction, and cublasGemmEx—contributors gain hands‑on insight into the math that powers modern inference.

Engineers and educators can fork the project, tweak paths for their CUDA toolchain, and submit pull requests, turning the repository into a living learning resource for building production‑ready inference services.