Building makemore Part 2: MLP

This lecture builds a multilayer perceptron (MLP) language model to overcome the limitations of simple bigram counting by using learned character embeddings and context windows to predict the next character in a sequence.

• Context limitation — simple models struggle because the number of required character combinations grows exponentially with the sequence length [01:04].

• Bengio paper approach — words are assigned vector embeddings in a shared space, allowing the model to learn relationships where similar items cluster together [03:04].

• Input structure — the code converts a chosen "block size" of previous characters into a set of numeric inputs for the neural network [11:23].

• Embeddings — the model uses a lookup table (matrix C) where each character gets a small, dense numeric vector; PyTorch indexing makes retrieving these vectors for entire batches very efficient [16:44].

• Network architecture — the neural network relies on:

  • Input layer (embedded characters) [18:54]
  • Hidden layer (fully connected with tanh non-linearity) [28:01]
  • Output layer (logits for the next character) [29:37]

• Loss function — standardizing on F.cross_entropy is preferred over manual calculations because it is mathematically stable, robust to extreme values, and optimized to run faster [33:57].

• Optimization — instead of calculating the gradient on the entire dataset, processing small, random subsets (mini-batches) allows for faster updates even if the gradient estimate is noisy [42:23].

• Learning rate tuning — finding the ideal speed for updates involves testing a range of values exponentially and plotting the loss to find the "valley" where the model converges reliably [50:08].

• Data splitting — to prevent the model from simply memorizing the input, the data is divided into three separate sets:

  • Training (for gradient descent) [54:36]
  • Development/Validation (for tuning hyperparameters) [55:14]
  • Test (for final performance evaluation) [55:51]

• How does F.cross_entropy handle large logit values compared to manual calculation?

• Why is it necessary to use separate datasets for training, development, and testing?