Lesson 21 of 38 · Written learning
Pretraining, checkpoints, and compute
Understand the next-token training loop and why student projects use small models.
A pretraining loop feeds a batch of token sequences to a model. At each position, the target is typically the following token. Cross-entropy measures how much probability the model assigned to those targets. The optimizer updates parameters, and the process repeats across many batches.
A loss curve is evidence, not a verdict
Track training and validation loss, save checkpoints, and record code, data, settings, and hardware. Learning-rate schedules, gradient clipping, and batch size influence stability. Throughput and GPU memory constrain the experiment. Scaling laws describe broad empirical relationships among data, model size, and compute; they do not provide a shortcut that makes frontier-scale training realistic on a student laptop.
- Start with a very small run to verify the pipeline before spending compute.
- A decreasing training loss does not prove useful or safe behavior.
- Compare experiments only when the data split and evaluation setup are controlled.
- Check licenses and storage policies for both training data and saved model weights.
Return to the Data engineering and pretraining stage · Suggested timing is flexible; move at your own pace.