Let's build the GPT Tokenizer
Andrej Karpathy · 2:13:34 · 2 years ago
Tokenization is a critical pre-processing stage that translates raw human text into sequences of numbers, but its underlying complexities and design choices—rather than the language model itself—are the root cause of many common issues like poor arithmetic, weak spelling, and language bias.
- Separate pipeline stage — Tokenization exists independently of the main model, using its own training data and algorithms to turn strings into integer sequences and back into text .
- Root cause of errors — Many performance problems often blamed on the neural network actually stem from the tokenizer .
- Spelling mistakes — Long, complex tokens contain too much information, making it difficult for the model to count or process individual letters .
- Poor arithmetic — Arbitrary number tokenization creates inconsistent representations, preventing the model from recognizing digits as distinct, sequential parts of a math problem .
- Language bias — Non-English text often results in larger, more "bloated" token sequences, wasting the model's limited context window compared to English .
- Byte Pair Encoding — This core algorithm compresses data by iteratively identifying the most frequent consecutive pair of tokens and merging them into a single new token .
- Vocabulary trade-offs — Increasing the number of tokens in a vocabulary allows for denser text representation but forces the model to maintain a larger, more computationally expensive embedding table .
- UTF-8 reliance — While Python uses Unicode code points, models prefer UTF-8 byte sequences because they avoid the wastefulness of fixed-length encodings like UTF-32 .
- GPT-4 efficiency — By optimizing whitespace handling and increasing vocabulary size, GPT-4 creates more efficient token representations than previous versions, allowing the model to process more context .
- Special delimiters — Markers like "end of text" are manually inserted into data streams to help the model recognize document or conversation boundaries .
How does vocabulary size influence the effectiveness of a language model? Why is Byte Pair Encoding preferred over using raw byte sequences?