Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling
Stanford Online · 1:03:20 · 2 days ago
Test-time compute scaling improves model performance without additional training by allocating more processing resources during the inference phase, often outperforming larger, pre-trained models on complex reasoning and coding tasks.
- Inference scaling — Increasing compute during the inference phase allows models to perform better without modifying existing parameters or conducting fine-tuning .
- Repeated sampling — Generating multiple responses for a single prompt follows a predictable power law, where solution coverage increases as the number of parallel samples scales .
- Hard problem tail — The effectiveness of sampling relies on a long tail of complex tasks; these are solvable by the model but only infrequently, necessitating high sample counts to capture successful outcomes .
- Verification gap — Large performance disparities exist between simple majority voting and oracle-level verification; isolating correct answers from candidates remains a hurdle in domains lacking automated checking tools .
- Reward mechanisms — Strategies to guide search include:
- Outcome models — Scoring final solutions based on correctness .
- Process models — Evaluating individual reasoning steps to provide granular feedback during generation .
- Archon framework — An inference-time architecture search method combines techniques like fusion, critic, ranker, and unit test generation to optimize reasoning; this approach improved pass-at-one accuracy by 14.1% over frontier models like GPT-4 and Claude 3.5 Sonnet .
Discussion Questions