Deep Dive into LLMs like ChatGPT
Andrej Karpathy · 3:31:23 · 1 years ago
Large language models function as sophisticated statistical engines that predict sequences of tokens rather than sentient entities. Because they rely on stochastic generation and are trained on internet data, they effectively serve as simulators of human behavior rather than infallible experts, necessitating a tool-first approach where users verify outputs and provide oversight for high-stakes tasks.
- Data preparation — Providers gather vast amounts of internet text and apply filtering for quality and safety before converting content into tokens, the fundamental atomic units processed by the model .
- Tokenization mechanics — Raw text is converted into discrete "tokens" using algorithms like Byte Pair Encoding, which compresses input data to fit within finite context windows .
- Neural network training — Models learn by iteratively predicting the next token in a sequence and adjusting billions of internal parameters to minimize error relative to the training data .
- Assistant creation — Raw base models, which merely complete text, are transformed into useful assistants through supervised fine-tuning on curated conversations to adopt helpful and harmless personas .
- Reasoning strategies — Reinforcement learning enables models to develop internal "chains of thought," allowing them to break down complex problems into manageable intermediate steps for higher accuracy .
- Tool integration — Models overcome limitations in internal memory by leveraging external tools, such as web search or code interpreters, effectively moving computational tasks to more reliable external processors .
- Inherent limitations — Models are prone to hallucinations and exhibit uneven performance across different domains, often failing at simple logical tasks despite succeeding at complex professional challenges .
Discussion Questions