Let's reproduce GPT-2 (124M)

Reproducing the GPT-2 (124M) model is achievable using PyTorch by implementing the architecture from scratch, applying standard initialization techniques, and utilizing modern software optimizations to reach performance levels comparable to the original model.

• Architecture — the model reuses the input embedding matrix as the final classification layer, which saves parameters and provides an inductive bias for semantic similarity [1:07:47].

• Initialization — weights are initialized using a normal distribution with a 0.02 standard deviation, with residual layers scaled by 1/sqrt(N) to maintain stable activation variance [1:14:36].

• Training setup — hyperparameters follow the GPT-3 paper, including AdamW optimization, gradient clipping to 1.0, and cosine learning rate decay [2:16:57].

• Throughput improvements — training speed increases significantly by implementing several techniques:

  • Precision — switching to TF32 and BFloat16 formats reduces memory load [1:25:32].
  • Compilation — torch.compile fuses operations to eliminate redundant GPU memory round trips [1:48:28].
  • Flash Attention — integrating this kernel avoids materializing the massive attention matrix [2:00:29].

• Scalability — distributed data parallel (DDP) allows training across multiple GPUs, while gradient accumulation simulates the larger batch sizes required for stable optimization [2:47:19].

• Data efficiency — using the FineWeb-EDU dataset provides high-quality tokens, enabling effective training with far fewer total tokens than the original GPT-2 release [3:12:41].

• What is the impact of weight tying on the model's parameter count?

• How does kernel fusion reduce memory traffic between the GPU and its memory?