The spelled-out intro to language modeling: building makemore

Building a character-level language model can be achieved by either direct statistical counting or neural network optimization, with both methods yielding identical probabilistic results.

• Bigram counting — raw occurrences of character pairs are tallied into a 2D torch tensor and normalized to create probability distributions for the next character [23:59].

• Model generation — sampling new names involves using torch.multinomial to randomly select the next character based on the learned probability values [30:13].

• Assessing quality — the model is evaluated using negative log likelihood, where lower values indicate that the model is successfully assigning high probabilities to the correct next character [56:53].

• Neural adaptation — a single linear layer transforms one-hot encoded inputs into log counts, which are then processed via a softmax function to output valid probabilities [1:10:46].

• Optimization — gradient-based learning improves the model by iteratively adjusting parameters to minimize the negative log likelihood loss [1:33:05].

• Weight regularization — incentivizing weights to remain near zero acts as a form of smoothing, which prevents the model from becoming overconfident [1:51:46].

• How do you expand this model to predict more than one character ahead?

• What is the relationship between raw counts and probability distributions in this context?