Andrej Karpathy created MicroGPT, a 200-line Python script that trains and runs a GPT model from scratch with no external libraries. The script demonstrates the core algorithms powering large language models like ChatGPT using just pure Python. The model trains on 32,000 human names, learning statistical patterns to generate plausible new names like 'kamon' and 'karai'.

At its foundation, the model converts text to numbers using a simple tokenizer that assigns unique integers to each character plus a special BOS (Beginning of Sequence) token. During training, the model slides through sequences one position at a time, predicting the next token based on context. The raw outputs (logits) get converted to probabilities through softmax, and cross-entropy loss measures prediction accuracy. The entire system uses backpropagation to compute gradients and update parameters.

The transformer architecture uses embeddings to represent tokens as 16-dimensional vectors, with separate position embeddings to capture sequence order. Attention mechanisms allow each token to gather information from previous positions through query-key-value operations. This interactive walkthrough breaks down each component visually, making the complex mathematics behind LLMs accessible to beginners.