Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL
Stanford Online · 1:12:38 · 2 days ago
Train-time scaling enables smaller models to achieve high reasoning accuracy by allocating computational resources to iterative self-improvement and reinforcement learning, rather than relying solely on increased model parameter counts. This loop allows models to refine their reasoning chains by learning from their own outputs and utilizing verification feedback.
- Train-time scaling — allocating more computation during training allows smaller models to match or exceed the accuracy levels typically observed in much larger systems .
- Self-improvement loop — training on model-generated outputs, filtered by correctness, lets a system iteratively refine its own logic without requiring massive human-labeled datasets .
- STaR methodology — generating rationales for problems where the model initially failed, by providing the correct answer as a hint, helps the system bootstrap stronger reasoning capabilities .
- GRPO efficiency — replacing standard reinforcement learning critic models with grouped, normalized rewards reduces memory requirements, enabling reinforcement learning on larger parameter counts .
- DAPO stability — mitigating training instability in long reasoning chains requires techniques such as asymmetric clipping, dynamic sampling, and token-level loss to prevent entropy collapse and uncontrollable output length .
- Consistency vs. intelligence — current methods generally improve the "majority-at-K" metric—meaning the model produces the correct answer more consistently—rather than fundamentally raising the "pass-at-K" ceiling or intelligence level .
How does GRPO save memory compared to standard reinforcement learning algorithms?