Lesson 15 of 38 · Written learning
Why attention helps language models
Connect the idea of context to the attention mechanism used in transformers.
A word's role depends on its context. In “the bank raised its rates,” nearby words help distinguish a financial institution from a riverbank. Older sequence models process tokens step by step, which can make distant relationships difficult to preserve and limit parallel computation during training.
Queries, keys, and values
Attention projects each token representation into three learned vectors. A query represents what a position is looking for; keys represent what other positions offer for matching; values carry the information that can be combined. Similarity between a query and keys produces weights, and the output is a weighted mixture of values.
scores = query @ key.T
weights = softmax(scores / sqrt(key_dim))
context = weights @ value- The projections are learned parameters.
- Attention weights are context-dependent, not a fixed dictionary of meanings.
- The model can relate tokens that are far apart, subject to its context and mask.
- The simplified equations omit batching, heads, masks, and implementation details.
Return to the Transformers and LLM architecture stage · Suggested timing is flexible; move at your own pace.