Stanford CS329A Self-Improving AI Agents | Part 4 | Learning from Feedback with Tools/Code

Language models improve in accuracy and reliability by integrating external feedback loops—such as real-world tool interaction, code execution outcomes, and rule-based self-critique—rather than relying solely on internal probability weights.

• Reasoning loops — Combining thinking and acting allows models to verify information using external tools, reducing reliance on internal memory [05:04].
• Task grounding — Using search or API calls provides real-world context for questions, which minimizes hallucination compared to standard models [03:50].
• Action sequences — By interleaving thought, action, and observation, the system creates interpretable decision traces that users can audit [06:15].
• Code optimization — Coding agents use pass/fail test results to iteratively debug and fix their own generated programs [28:55].
• Separated testing — Using public tests for immediate feedback and private tests for model evaluation prevents the system from simply memorizing answers [31:56].
• Rule-based critique — Written principles guide models to identify and revise harmful or unethical content without requiring ongoing human oversight [48:04].
• Model alignment — AI-generated feedback acts as a substitute for human labeling, training the system to prioritize safety while maintaining helpfulness [51:47].

Discussion Questions

• How does the ReAct framework manage uncertainty when the environment returns noisy or misleading feedback?
• Why is separating public and private test sets important when training coding agents to avoid memorization?