Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 9: Stochastic Dyn. Program

Stochastic dynamic programming allows systems to handle environmental uncertainty by making optimal decisions that account for random fluctuations in state transitions. This approach extends deterministic control methods to probabilistic settings, enabling computers to calculate effective actions even when future outcomes are not guaranteed.

• Markov decision processes — The framework models systems where external disturbances affect forward motion, requiring policies to map current states to optimal control actions [00:42].
• Memoryless dynamics — The system assumes that random disturbances depend only on the current state and chosen control, effectively disregarding previous history [04:05].
• Expected value — Because disturbances make the cost a random quantity, the objective is to minimize the average cost across all potential future outcomes [06:42].
• Recursive solution — Algorithms solve the problem by moving backward from the final time stage, calculating the best trade-off between current costs and future consequences [20:08].
• Noisy control systems — For linear systems with random disturbances, the optimal control remains a linear feedback loop, though the overall cost rises relative to the noise magnitude [52:31].
• Infinite horizons — Systems operating over indefinite timeframes use a discount factor to ensure the total cumulative reward calculation remains finite and convergent [55:40].
• Q-function utility — This method represents the expected cumulative reward of taking a chosen action, which makes finding the best action easier when transition details are unknown [1:10:32].

How does the discount factor impact the reward calculation in an infinite-horizon model? How do Q-functions facilitate determining an optimal control action when transition dynamics are unknown?