Lesson 7 of 38 · Written learning
Derivatives, gradients, and learning
See how a model can use an error signal to improve its parameters.
A function maps an input to an output. A derivative describes how a small change in an input affects the output. With many inputs, a partial derivative measures sensitivity to one input while holding the others fixed. The gradient collects those partial derivatives into one vector.
From a loss to a parameter update
A model makes a prediction and a loss function measures how far that prediction is from a target. The chain rule links derivatives through a sequence of operations, letting backpropagation efficiently compute how each parameter contributed to the loss. Gradient descent then changes parameters by a small amount opposite the gradient.
parameter = parameter - learning_rate * gradient- A learning rate that is too large can make updates overshoot.
- A learning rate that is too small can make useful improvement very slow.
- The gradient depends on the current data batch and current parameter values.
- Backpropagation computes derivatives; the optimizer decides how to use them.
Return to the Mathematics for LLMs stage · Suggested timing is flexible; move at your own pace.