Stanford CS229 Machine Learning | Spring 2026 | Lecture 16: Basic Concept in RL, Policy Gradient

Modern transformer architecture and large language models (LLMs) rely on optimizations to balance high parameter counts with efficient computation and memory usage. Methods such as grouping attention heads, activating sub-networks sparsely, and utilizing instruction-based training enable these models to maintain high performance without requiring prohibitive hardware resources.

• Memory usage — storing all key and value vectors per head creates massive GPU memory requirements, which restricts batch size and limits overall hardware utilization [05:47]

• Grouped Query Attention — this optimization shares a smaller, fixed set of keys and values across multiple query heads, significantly lowering the total storage needed during text generation [07:41]

• Sliding window — restricting attention to a fixed number of recent tokens changes computation from quadratic to linear, addressing memory bottlenecks in long sequences [27:09]

• Mixture of Experts — using a routing module to activate only a small subset of available sub-networks allows for a massive total parameter count while keeping active computation low [35:21]

• In-context learning — models perform tasks by analyzing examples or descriptions provided within the input prompt, which shifts output behavior without altering any internal weights [52:36]

• Instruction tuning — training on pairs of task descriptions and answers aligns the model to follow user formatting, effectively improving its utility for tasks without retraining the entire system [1:04:46]

• How does the routing module determine which experts are activated?

• What characterizes the difference between in-context learning and instruction tuning?