Stanford CS229 Machine Learning | Spring 2026 | Lecture 3: Weighted Least Squares

This lecture establishes the framework for classification by moving from linear regression to probabilistic modeling. By using the principle of maximum likelihood, researchers can derive logistic regression and select appropriate optimization methods based on dataset size and computational constraints.

• Classification goal — predict discrete outcomes rather than continuous numbers [00:22].
• Probabilistic model — treat data generation as a process involving an underlying hidden variable plus a noise term [05:49].
• Gaussian assumptions — define noise as unbiased, independent, and identical samples following a normal distribution [07:26].
• Maximum likelihood — select parameters that maximize the probability of observed data by transforming the likelihood into a log-based sum [27:20].
• Least squares equivalence — show that minimizing the squared error is mathematically identical to maximizing the likelihood of a model with normal noise [28:25].
• Logistic regression — apply a link function to linear results, mapping them to a 0-1 probability range to enable binary classification [38:28].
• Newton’s method — employ second-order derivatives for fast convergence, though this approach becomes computationally heavy as dimensions increase [53:37].
• Computational efficiency — favor stochastic gradient descent for large-scale tasks because it requires far less processing power per update than second-order methods [56:26].

How does the assumption of Gaussian noise impact model performance? What is the trade-off between using Newton's method and stochastic gradient descent?