Lesson 18 of 38 · Written learning
Inference and decoding strategies
Understand how token probabilities become a visible response.
At inference time, the model processes the available prompt and generated tokens, then produces logits for the next token. Greedy decoding chooses the most likely token each time. Sampling draws from a distribution, allowing varied outputs. Temperature reshapes the distribution: lower values generally concentrate probability on high-scoring options, while higher values make sampling more varied.
Top-k, top-p, and stopping
Top-k sampling keeps a fixed number of candidates. Top-p sampling keeps the smallest set whose cumulative probability reaches a chosen threshold. These choices trade off diversity and predictability. A stop token, stop string, or maximum token limit ends generation. Context limits include both input and generated tokens, depending on the system.
- Different settings can produce different answers from the same model.
- A low temperature does not guarantee correctness.
- Longer outputs use more time and often more compute.
- Record the model version, prompt template, and decoding settings when comparing results.
Return to the Transformers and LLM architecture stage · Suggested timing is flexible; move at your own pace.