Stanford CS329A Self-Improving AI Agents | Part 5 | Planning and Multi-Step Reasoning
Stanford Online · 1:14:55 · 2 days ago
LLMs can solve complex, multi-step problems more effectively by using search-based planning (LATS), parallelizing independent reasoning steps (SPRINT), and offline reinforcement learning (SWiRL). These approaches improve reasoning accuracy and reduce computational costs by optimizing how models generate, evaluate, and execute sub-tasks.
- LATS framework — integrates Monte Carlo Tree Search (MCTS) into reasoning paths, combining model-based evaluation with self-consistency scores to navigate decision trees for complex tasks .
- Tree expansion — guides the search by calculating a value function for nodes, allowing the model to balance exploration and exploitation similar to traditional game-playing algorithms .
- Action selection — relies on UCT (Upper Confidence bounds applied to Trees) to prioritize promising paths while visiting unexplored options to avoid getting stuck in suboptimal sequences .
- SPRINT method — identifies independent segments in a reasoning chain and trains models to execute them in parallel, significantly lowering sequential token output and total inference time .
- Data annotation — uses an external model to rewrite existing reasoning traces into a format containing explicit plan and execution blocks, which then serves as training data .
- Parallel execution — leverages structural independence, enabling the model to generate multiple plans simultaneously rather than waiting for each sequential step to finish .
- SWiRL approach — generates offline synthetic trajectories and trains models using multi-step reinforcement learning, teaching them to select tools and reason without requiring active tool execution during the training phase .
- Process rewards — utilizes an LLM to assign quality scores to reasoning steps, enabling the model to learn effective decomposition without needing to verify tool outputs during the optimization process .
- Generalization — demonstrates that models trained on one domain, such as math, can transfer reasoning capabilities to entirely different domains like search-based question answering .