Stanford CS229 Machine Learning | Spring 2026 | Lecture 6: Dataset Split, ML Advice

The fundamental challenge in machine learning is managing model complexity to ensure it performs well on unseen data, a balance traditionally managed through the bias-variance trade-off but complicated by modern over-parameterized architectures.

• Bias and variance — Models encounter a trade-off between bias, where the model is too simple to represent the data, and variance, where the model is too sensitive to random noise in the training set [00:45].

• Error components — Prediction error breaks down into three factors: intrinsic noise, systematic bias (the distance between the model average and the truth), and variance (how the model fluctuates based on the training sample) [32:20].

• Regularization utility — Techniques like Ridge regression minimize variance by adding a penalty for large weights, which trades a controlled amount of bias for a more stable model [42:40].

• Double descent — Contrary to classical intuition, highly complex models can exhibit improved test performance when their parameter count far exceeds the number of data points [1:04:25].

• Benchmarking validity — Replicating established datasets suggests that model rankings remain stable over time, indicating that overfitting to public test sets is less severe than previously assumed [1:07:55].

• Compute efficiency — Algorithms like Hyperband optimize parameter tuning by running multiple configurations and incrementally discarding poorly performing ones, focusing resources on the most promising candidates [1:13:28].

• How does data augmentation serve as a form of regularization?

• What distinguishes the role of dev sets from test sets in model evaluation?