Stanford CS229 Machine Learning | Spring 2026 | Lecture 4: Exponential Family, GLMs classification

The exponential family provides a unified mathematical framework that generalizes linear models, enabling consistent learning and inference procedures—such as gradient descent and maximum likelihood estimation—to be applied across diverse data distributions and error types.

• Unifying Framework — By casting statistical models into a standard functional form, inference tasks like computing expectations or variances become automatic operations [01:39].
• Core Components — The framework relies on three essential elements to define a probability distribution:
  • Sufficient statistic — The measurement captured about the target variable, which often simplifies to an identity function [04:42].
  • Base measure — A term that acts as a normalization factor and remains independent of the distribution's parameters [05:04].
  • Log partition function — A central term that normalizes the distribution, where differentiation yields the mean and variance [05:15].
• Model Optimization — Generalized linear models link input features to the natural parameter via weights, allowing practitioners to solve for parameters using stochastic gradient descent [01:42].
• Softmax Utility — This technique generalizes logistic regression for multiclass scenarios by normalizing the exponential of linear scores, producing a probability distribution across categories [40:44].
• Label Smoothing — Introducing controlled noise into target labels acts as a form of regularization, preventing models from overfitting and curbing overconfidence in training data [1:05:27].
• Geometric Interpretation — While low-dimensional visualizations suggest linear models are limited, high-dimensional spaces provide sufficient volume to allow for effective linear separation of data points [46:16].

How does the log partition function generate the mean and variance of a distribution? How does label smoothing mitigate overfitting during model training?