Inspired by minGPT for home experiments. Requires CUDA 13.4; build script is Windows only.

Tiny byte-level GPT training in CUDA C++. MIT. Build Linux/Nix OS: nix-build -o build/nix-result Windows, Visual Studio 2022 C++ tools, and CUDA 13.4: Cuda Arch is the GPU compute capability from NVIDIA's CUDA GPU list. Run. \build\turbogpt.exe --data hn1g.txt --log-to runs/ctx4.

The run stores its checkpoint at runs/ctx4/ctx4.pt, containing model, optimizer, scheduler, and trainer state. Use --load CHECKPOINT.pt to resume it. runs/ctx4/report.json is derived from the log directory. Logs are Tensor Board-compatible: one report per batch, capped at 8Mi reports, and flushed with periodic or final checkpoints. Result hn1g after 1.5G training tokens: 2.5295 BPB.