The spelled-out intro to neural networks and backpropagation: building micrograd
Andrej Karpathy · 2:25:52 · 3 years ago
Training a neural network is essentially building a mathematical expression and using backpropagation—a recursive application of the chain rule—to measure how every parameter affects the final outcome, then iteratively adjusting those parameters to minimize loss.
- Core engine — micrograd tracks scalar values to demonstrate the math behind backpropagation without the added complexity of high-performance tensor libraries .
- Computation graph — every value object stores its own data, gradient, and the set of parent nodes that created it, allowing the system to remember how it calculated the final result .
- Chain rule — this calculus method allows for calculating how an output changes by multiplying the local derivative of each operation by the upstream gradient .
- Backward propagation — the engine automates the process by:
- performing a topological sort to order nodes from output to input .
- iterating through these nodes in reverse, applying the chain rule at every step to propagate gradients .
- Gradient accumulation — failing to add gradients together (rather than overwriting them) when a node is used multiple times in a graph leads to incorrect math .
- Zero grad — a critical training step involves manually resetting all parameter gradients to zero before starting a new backward pass to prevent stale values from accumulating .
- Neural architecture — models are constructed as layers of neurons that compute a weighted sum of inputs plus a bias, passed through a non-linear "activation function" like tanh .
- Optimization — training involves adjusting weights by a small step (the learning rate) in the opposite direction of the gradient to reduce the total loss .
Questions: