Stanford CS229 Machine Learning | Spring 2026 | Lecture 7: Neural Networks 1 (Architecture)
Stanford Online · 1:20:27 · 5 days ago
Neural networks and deep learning methods enable the modeling of complex, non-linear relationships in data by stacking layers of computations, which are optimized efficiently using stochastic sampling methods rather than exhaustive full-dataset calculations.
-
Nonlinear modeling — Networks provide a way to map inputs to outputs that go beyond straight lines by applying an activation function to linear combinations of data .
-
Loss functions — Training minimizes error metrics like mean squared error for continuous prediction tasks or cross-entropy for categorical classification .
-
Gradient descent — Learning involves updating parameters based on the direction that reduces loss; however, calculating this gradient for every data point is too costly for large datasets .
-
Stochastic optimization — Sampling small batches or single items to estimate the gradient provides an efficient alternative, where the inherent noise in the path eventually averages out to the correct descent direction .
-
High-dimensional geometry — In complex models, the landscape tends to allow for downward paths in almost all directions, making the risk of getting trapped in poor local minimums lower than in simple models .
-
ReLU activation — This common primitive sets negative inputs to zero and leaves positive inputs unchanged, acting as the nonlinear "switch" that allows stacking of layers .
-
Layer composition — Stacking layers by performing matrix multiplication followed by an activation function allows the network to learn intermediate features, such as abstract qualities in housing data, before reaching a final prediction .
-
Residual connections — Adding the input of a layer block back to its output forces the network to model only the difference—or residual—between the current state and the target, which can simplify the learning task .
-
Normalization — Adjusting the output of layers to keep values within a consistent range prevents numerical explosions and stabilizes the training process .
-
How does the batch size used during optimization influence GPU resource efficiency?