Building makemore Part 3: Activations & Gradients, BatchNorm

Deep neural networks are often fragile during training due to improper initialization, which leads to vanishing gradients or saturated neurons; Batch Normalization provides a solution by standardizing activations, allowing for more stable training and reliable performance even in deeper architectures.

• Initial loss spikes — Neural networks often start with an incorrectly high loss because parameters are improperly scaled, causing the model to be "confidently wrong" at the start [04:34].

• Vanishing gradients — Activation functions like tanh can cause gradients to shrink to zero when inputs are too large, which effectively kills the learning process [17:46].

• Dead neurons — When neurons become permanently saturated (e.g., a ReLU neuron always outputting zero), they stop responding to inputs and suffer permanent "brain damage" [21:46].

• Kaiming initialization — Scaling weights by the square root of the number of input connections helps preserve variance and prevents signals from exploding or vanishing [30:42].

• Batch Normalization — This technique standardizes internal activations to be roughly Gaussian, which stabilizes training across deep architectures [40:50].

• Spurious biases — Because Batch Normalization centers data, adding a bias term in the preceding linear layer is redundant and unnecessary [1:02:04].

• Inference calibration — After training, the parameters used for Batch Normalization can be "folded" into the preceding linear layer, allowing for clean, single-example predictions [54:46].

• Diagnostic tools — Tracking the ratio of updates to parameter data is a reliable method to verify if the learning rate is well-tuned [1:40:09].

• How does Batch Normalization impact inference compared to training?

• Why do neural networks require careful initialization to maintain activation variance?