Lesson 13 of 38 · Written learning
Language models and next-token prediction
Understand how simple n-gram models lead to the central training objective of an LLM.
An n-gram model estimates a next token from a short window of previous tokens. It is easy to understand and can provide a baseline, but it has limited memory and struggles with rare contexts. Neural language models learn distributed patterns that can share information across related contexts.
Learning from shifted sequences
For next-token training, a text sequence is arranged so each position predicts the following token. The model receives earlier tokens and produces a distribution over possible next tokens. Repeating this across many sequences trains the model to capture regularities in the data. This objective supports generation, but it does not directly teach a model to verify claims or follow every instruction.
- A character-level model uses characters as units; a token-level model uses a tokenizer's units.
- Short context windows make simple models cheap but limit longer dependencies.
- Self-supervised targets can be derived from the text without human labels for each next token.
- Training data influences the patterns the model can reproduce, including its gaps and biases.
Return to the NLP and language-model fundamentals stage · Suggested timing is flexible; move at your own pace.