Stanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning

Diffusion models function by learning to reverse the noise addition process, while foundation models leverage a two-stage paradigm consisting of massive-scale pre-training followed by efficient task adaptation, with Low-Rank Adaptation (LoRA) serving as the primary method for enabling multi-tenant model deployment.

• Training objective — Diffusion models optimize parameters by maximizing the log likelihood of the original data, simplified using a variational lower bound into a mean-squared error problem [04:00].

• Noise prediction — The training process is structured as a regression task where the network attempts to estimate the noise added at each time step rather than generating images directly [26:13].

• Paradigm shift — Current development workflows are divided into two distinct phases:

  • Pre-training on massive, diverse, unlabelled datasets [40:07].
  • Adaptation to downstream tasks without requiring exhaustive label collection [39:47].

• Linear probing — An adaptation technique where the base model parameters remain frozen, and a new linear head is trained to map extracted features to target outputs [50:12].

• Low-Rank Adaptation (LoRA) — This method constrains the degrees of freedom during weight updates by reparameterizing matrices as the product of two smaller, low-rank matrices [1:05:00].

• Deployment efficiency — LoRA allows infrastructure providers to serve many users concurrently by maintaining one shared base model in memory while swapping only the lightweight, user-level adapters [1:13:54].

• How do linear probing and fine-tuning differ regarding parameter optimization?

• What is the primary operational advantage of using LoRA for model deployment?