Large AI models are powerful but expensive and resource-intensive. Knowledge distillation offers a solution by training smaller "student" models to mimic the behavior of larger "teacher" models. This process allows smaller models to achieve comparable or even superior performance on specific tasks, making advanced AI accessible on resource-limited devices.

Distillation differs from compression methods like quantization, which simply shrink an existing model. Instead, distillation creates a distinct, smaller model. The key advantage lies in the "soft labels" provided by the teacher model. Unlike single "hard labels" from raw data, soft labels are probability distributions that reveal the relationships between different options, offering a richer training signal.

There are three main distillation methods: output distillation (matching final probabilities), feature distillation (matching internal representations), and synthetic data distillation (training on teacher-generated examples). Synthetic data distillation is currently the most common due to broader accessibility. These techniques are vital for deploying efficient AI, with examples like Google's Gemma models utilizing distillation during their training. This approach enables AI to run faster, cheaper, and even on-device.