Stanford CS336 Language Modeling from Scratch | Spring 2026 | Guest Lecture: Dan Fu

Effective model serving requires deep control over low-level hardware operations to overcome memory and compute bottlenecks, rather than relying solely on high-level architecture changes. By optimizing GPU "kernels"—the programs that actually execute mathematical operations—engineers can significantly increase efficiency and enable new model architectures.

• Inference importance — The process of serving models converts electricity into intelligence and acts as the crucial engine for AI applications [05:14].

• Prefill vs. Decode — Prefill operations are compute-heavy, whereas decoding is memory-bandwidth heavy, leading many systems to run these on separate hardware [14:27].

• KV cache management — Large production systems store key-value (KV) caches across GPU, CPU, and disk to maintain context without recomputing, similar to traditional operating system paging [18:07].

• Mega-kernel optimization — By fusing multiple operations into a single kernel, developers can drastically reduce idle time and achieve near-peak GPU performance [37:34].

• Implementation difficulty — Creating these kernels is labor-intensive, often requiring a year of effort from a skilled engineer to support just a few hardware configurations [47:06].

• Parse architecture — This approach uses loop-transformers, where tokens pass through the same blocks multiple times to boost performance without adding parameters [42:22].

• Training stabilization — Standard loop models often suffer from training failures; reparameterizing matrices in the Parse model prevents these unstable loss spikes [51:23].

• How does the Parse model balance the trade-off between recurrent loops and increased parameter counts?

• What are the primary bottlenecks when scaling KV cache management across different storage tiers?