Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA

Reinforcement learning algorithms like Policy Gradient, PPO, and GRPO allow models to optimize reasoning capabilities by treating token generation as a sequence of actions. By applying rewards to the final output rather than intermediate steps, models can discover effective chains of thought independently, provided the training process starts with a baseline capability that generates at least some positive results.

• Reward weighting — Policies improve by increasing the likelihood of high-reward outcomes while decreasing the probability of low-reward ones [03:32].
• Baseline subtraction — Calculating the difference between rewards and a reference value based on the state reduces noise and variance during training [18:05].
• Importance sampling — This technique allows models to reuse data from previous policies by adjusting for the probability differences between old and new actions [29:30].
• Constraint clipping — Limiting how much the model changes its policy ensures updates remain stable, preventing the model from over-adjusting when it already generates high-quality outcomes [40:55].
• Reasoning optimization — LLM "thinking" is treated as a sequence of states and actions, where reinforcement is applied only once a correct final answer is reached [1:03:20].
• Initialization requirement — Effective training requires starting with a baseline capability that yields non-zero rewards, often achieved through initial supervised instruction tuning [1:13:46].

How does the choice of reward calculation impact the stability of training? What are the limitations of using a language model as a reward function compared to ground-truth verification?