Lesson 17 of 38 · Written learning
Transformer blocks and model families
Assemble attention with feed-forward layers and compare encoder, decoder, and encoder–decoder designs.
A typical transformer block contains an attention operation and a position-wise feed-forward network. Residual connections add an input back to a sublayer's output, helping information and gradients move through depth. Normalization stabilizes activations. Implementations vary in ordering and detail, so read the model documentation rather than assuming every block is identical.
Three common architecture families
Encoder-only models build contextual representations of input text and are often used for understanding tasks. Decoder-only models predict one token after another and power many generative systems. Encoder–decoder models encode an input and then generate an output, a useful pattern for tasks such as translation. The architecture suggests a training and use pattern, not a guarantee of task quality.
- Positional information lets a model distinguish order; implementations may use learned or structured positional methods.
- Parameter count is one measure of model size, but memory, data, and compute also matter.
- The context window limits how much input the model can consider in one pass.
- A tiny transformer project teaches the components; it does not reproduce the capability of a frontier model.
Return to the Transformers and LLM architecture stage · Suggested timing is flexible; move at your own pace.