Stanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA

Reinforcement learning (RL) facilitates sequential decision making where an agent learns to interact with an environment to maximize cumulative rewards, utilizing the Markov Decision Process (MDP) framework and policy gradient methods to optimize behavior without requiring labeled training data.

• Sequential decision making — Decisions affect future states, requiring agents to balance immediate rewards against long-term outcomes [01:42].
• MDP structure — Formal modeling requires defining the state space, available actions, transition probabilities, reward mechanisms, and a discount factor for future values [07:26].
• Policy function — Agents use a policy, or a mapping from states to actions, to determine behavior; stochastic policies are often preferred during training to maintain continuous parameter updates [34:17].
• Bellman equation — This recursive method computes value functions by breaking down the total expected return into an immediate reward plus the discounted future value [43:47].
• Policy gradient — Parameters are optimized by calculating the gradient of expected returns, which uses sampled trajectories to reinforce successful actions instead of requiring ground-truth labels [56:22].

How does the discount factor impact the calculation of long-term rewards in an MDP? How does the policy gradient method account for the lack of ground-truth labels during training?