Stanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification

Verification methods for AI agents have shifted from checking final answers to evaluating individual reasoning steps, enabling more reliable outputs through automated labels and ensemble-based scoring systems.

• Outcome-based rewards — Systems evaluate only the final result, treating the reasoning path as a black box that is either right or wrong [03:23].
• Process-based supervision — Models assign a score to each individual reasoning step, allowing them to pinpoint the exact moment a calculation deviates from the correct path [22:29].
• Automated labeling — Techniques like Math-Shepherd remove the need for human input by generating multiple potential continuations from a single step to statistically predict its success [39:40].
• Ensemble verification — Combining outputs from multiple different verifiers using weighted averages creates a more robust reliability signal than relying on any single model [53:47].
• Model distillation — Complex, large-scale verification systems can be condensed into much smaller, lightweight units that retain nearly all accuracy, drastically lowering computing requirements [1:04:58].

How does the use of process-based rewards differ from outcome-based rewards regarding the management of false positives?

What are the benefits and drawbacks of using ensembles of verifiers compared to single-model approaches?