Building makemore Part 5: Building a WaveNet

This hierarchical, tree-like architecture improves character-level language modeling by progressively fusing information, outperforming previous flat model designs after correcting dimension-handling bugs in the normalization layers.

• Code refactoring — Simplified the forward pass by building custom embedding and flattening modules, then organizing all layers into a sequential container for cleaner execution [12:15].

• Hierarchical architecture — Replaced the single-hidden-layer bottleneck with a tree-like design, which fuses input characters in pairs to process context more deeply [19:00].

• Batch normalization fix — Updated the normalization layer to handle 3D inputs correctly, ensuring the running mean and variance calculations reduce over the correct dimensions [43:06].

• Convolutional efficiency — Demonstrated how sliding linear filters over input sequences achieve the same goals as the hierarchical structure but allow for faster computation and parameter reuse [51:23].

• Development workflow — Followed a standard practice:

  • Prototyping layer shapes in notebooks
  • Validating tensor operations before committing code to the main repository [54:05].

• What does the hierarchical structure aim to achieve in a language model?

• Why are convolutional layers more efficient than manual hierarchical linear layers?