Lesson 8 of 38 · Written learning
Probability, logits, and loss
Connect model scores to probabilities and to the objective used during training.
A probability distribution assigns non-negative values that add to one across possible outcomes. A model often produces one raw score, called a logit, for each possible token. Softmax exponentiates and normalizes those scores. Increasing one logit tends to increase that option's probability relative to the others.
Why logarithms appear in loss
Cross-entropy compares a target distribution with the distribution predicted by a model. For a single correct token, the loss is the negative log probability assigned to that token. A confident correct prediction has low loss; assigning tiny probability to the correct token has high loss. Entropy describes uncertainty in a distribution, while expectation describes an average weighted by probabilities.
- Softmax probabilities depend on all candidate logits, not on one score alone.
- Numerically stable implementations subtract the largest logit before exponentiating.
- Sampling from probabilities is not the same as choosing the highest-probability token.
- Statistics and test splits help determine whether observed performance generalizes beyond the data used for learning.
Return to the Mathematics for LLMs stage · Suggested timing is flexible; move at your own pace.