Large Language Models (LLMs) learn to be helpful through a multi-stage process. The first stage, pretraining, involves reading vast amounts of text to build a general understanding of language and world knowledge. This provides raw capability but lacks specific direction.

The second stage is supervised fine-tuning (SFT), where models are trained on example pairs of instructions and good responses, teaching them to imitate desired outputs. However, SFT struggles when questions have multiple valid answers, as it teaches imitation rather than weighing trade-offs.

The third stage focuses on teaching preferences, crucial for nuanced helpfulness. This is achieved through comparison data, where humans or AI judge which of two responses is better. Two primary methods utilize this data: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). RLHF trains a separate reward model to score responses and then uses reinforcement learning to optimize the main model against this scorer, involving multiple models and a complex pipeline. DPO, a simpler and more direct approach, optimizes the model directly on the comparison pairs in a single training step, making it more efficient.