Lesson 16 of 38 · Written learning
Scaled attention, masks, and multiple heads
See why attention scores are scaled and how masking controls which positions a token can use.
Dot products tend to grow in magnitude as vector dimensions grow. Scaled dot-product attention divides the query-key scores by the square root of the key dimension before applying softmax. This helps keep the distribution from becoming extremely peaked too early in learning.
A causal view of the sequence
In autoregressive training, position four may use positions one through four, but not positions five onward. A causal mask blocks future positions before normalization. Multi-head attention runs several learned attention patterns in parallel, then combines their outputs. Heads can specialize, but they do not map neatly to human-labeled concepts by default.
- Causal masking enforces the direction of next-token prediction.
- Padding masks can prevent attention to filler positions in batches.
- Multiple heads increase the model's ability to represent different relationships.
- A shape check for the score matrix often catches mask and batch errors early.
Return to the Transformers and LLM architecture stage · Suggested timing is flexible; move at your own pace.