Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning

Large language models function as auto-regressive systems that predict the next token in a sequence using a transformer architecture. This process relies on converting text into numerical tokens, modeling conditional probability distributions via neural networks, and utilizing attention mechanisms to manage dependencies between tokens while balancing computational efficiency.

• Tokenization function — Models operate on numerical sequences rather than raw text, so input is converted into discrete sub-word units using a predefined vocabulary [02:01].

• Sub-word benefits — Breaking rare words into smaller, familiar units allows the model to leverage existing knowledge of linguistic patterns, significantly increasing training efficiency [04:03].

• Numerical mapping — Each vocabulary unit is assigned a unique numerical ID, transforming arbitrary text into a sequence of integers for processing [07:05].

• Auto-regressive logic — The system estimates sequence probability by decomposing the joint distribution into a product of conditional probabilities, where each prediction depends on all preceding tokens [12:30].

• Softmax distribution — Neural networks output logits that are passed through a softmax function to generate a probability distribution across the entire vocabulary for the next potential token [18:53].

• Temperature control — Dividing logits by a temperature parameter scales the distribution, allowing users to shift the model's output between deterministic choices and stochastic, diverse generation [29:15].

• Attention mechanism — To manage token dependencies, the model computes interactions between "query," "key," and "value" vectors to determine how much focus each token should place on others in the sequence [47:46].

• Causal masking — During auto-regressive generation, the model applies a mask of negative infinity to future token positions in the attention matrix, preventing the system from accessing information it is currently trying to predict [1:03:54].

• Parallel heads — Multiple attention heads run in parallel to capture different types of correlations before the results are concatenated and projected into a final vector [1:08:27].

• Scaling bottlenecks — Standard attention mechanisms scale quadratically ($T^2$) with sequence length, which creates major computational and memory barriers when processing extremely long inputs [1:14:33].

• How does the temperature parameter influence the probability distribution of generated tokens?

• Why is causal masking necessary in an auto-regressive transformer model?