Andrej Karpathy
Summaries
Let's build GPT: from scratch, in code, spelled out.
Building a functional GPT-style language model involves constructing a neural network that uses self-attention to allow tokens to communicate with one another, iteratively proce...
The spelled-out intro to neural networks and backpropagation: building micrograd
Training a neural network is essentially building a mathematical expression and using backpropagation—a recursive application of the chain rule—to measure how every parameter af...
The spelled-out intro to language modeling: building makemore
Building a character-level language model can be achieved by either direct statistical counting or neural network optimization, with both methods yielding identical probabilisti...
Building makemore Part 2: MLP
This lecture builds a multilayer perceptron (MLP) language model to overcome the limitations of simple bigram counting by using learned character embeddings and context windows ...
Building makemore Part 3: Activations & Gradients, BatchNorm
Deep neural networks are often fragile during training due to improper initialization, which leads to vanishing gradients or saturated neurons; Batch Normalization provides a so...
Building makemore Part 4: Becoming a Backprop Ninja
Manually implementing backpropagation on tensors is an essential exercise that transforms neural network optimization from a "magical" black box into a transparent, mechanical p...
Building makemore Part 5: Building a WaveNet
This hierarchical, tree-like architecture improves character-level language modeling by progressively fusing information, outperforming previous flat model designs after correct...
![[1hr Talk] Intro to Large Language Models](/_next/image?url=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzjkBMFhNj_g%2Fhq2.jpg&w=3840&q=70)
[1hr Talk] Intro to Large Language Models
Large language models (LLMs) function like the kernel of an emerging operating system, using next-word prediction to coordinate memory, external tools, and reasoning, while faci...
Let's build the GPT Tokenizer
Tokenization is a critical pre-processing stage that translates raw human text into sequences of numbers, but its underlying complexities and design choices—rather than the lang...
Let's reproduce GPT-2 (124M)
Reproducing the GPT-2 (124M) model is achievable using PyTorch by implementing the architecture from scratch, applying standard initialization techniques, and utilizing modern s...
Deep Dive into LLMs like ChatGPT
Large language models function as sophisticated statistical engines that predict sequences of tokens rather than sentient entities. Because they rely on stochastic generation an...
How I use LLMs
LLMs function as highly compressed, probabilistic versions of internet data that predict the next piece of text, rather than intelligent beings. To get the best results, users m...