Training large language models often crashes at the final layer due to the Cross-Entropy Loss. The logit bottleneck creates a massive memory spike, as projecting hidden states into a huge vocabulary space can demand over 80GB of VRAM for intermediate tensors, causing Out-of-Memory errors before training completes.

Standard fixes like gradient accumulation or activation checkpointing slow training significantly. A custom Triton kernel offers a smarter path by fusing the linear projection and loss calculation. This approach tiles computations to avoid materializing the full logit matrix, slashing peak memory usage by 84% without the usual performance trade-offs.

The kernel derives math for a fused forward and backward pass, computing key values like the target logit and log-sum-exp iteratively. While this implementation prioritizes educational clarity over production speed, it demonstrates how low-level optimization can solve critical bottlenecks. Future work aims to match the efficiency of libraries like Unsloth and Liger-Kernel.