Stanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon Tasks
Stanford Online · 1:15:17 · 3 days ago
AI agent performance is advancing rapidly in task duration and output quality, yet these systems remain unreliable for complex, multi-step work without human oversight. While models can complete longer tasks, they frequently encounter failure modes like poor planning and insufficient context awareness, effectively functioning as low-context assistants rather than autonomous experts.
- Task duration — Agent capability has progressed from managing 2-second atomic actions in 2019 to roughly 59-minute operations in 2025 .
- Capability growth — Measured by time-horizon metrics, agent capacity doubles roughly every seven months, though high reliability frequently plateaus at 50 percent .
- Reliability gap — Achieving an 80 percent success rate requires significantly shorter tasks, often lasting only 8 to 10 minutes, demonstrating a significant drop-off from 50 percent reliability levels .
- Context dependency — AI performance mimics low-context contractors; model speed and accuracy increase drastically when humans provide the necessary background information or project architecture .
- Economic value — Benchmarks comparing models to industry experts show a linear trend where models increasingly achieve parity in routine digital work, such as document formatting and data entry .
- Common failure modes — Frequent errors include poor planning, incorrect tool selection, premature task abandonment, and falling into repetitive action loops .
- Research synthesis — Current models fail to retrieve foundational papers or verify factual claims consistently, achieving low success on academic literature reviews despite generating fluent text .
- Human collaboration — Effective deployment relies on humans performing the architecture and prioritization, leaving the execution of well-defined steps to the model .