Lesson 24 of 38 · Written learning
LoRA, quantization, and preference tuning
Compare adaptation methods and understand what preference optimization is trying to change.
Full fine-tuning updates many or all model parameters and can require substantial memory. LoRA adds small trainable low-rank matrices to selected layers, leaving most original weights frozen. QLoRA combines low-bit quantization of the base weights with trainable adapters to reduce memory further. Quantization trades numerical precision for storage and speed; its effect depends on the model and task.
Preference optimization
Preference data compares responses, for example indicating which of two answers is preferred. RLHF commonly uses a learned reward model and reinforcement-learning steps. Direct Preference Optimization (DPO) uses preference pairs in a different optimization objective. These techniques can shape response preferences, but preference labels can be inconsistent and do not automatically make an answer true or safe.
- Use prompting when a clear instruction is enough.
- Use retrieval when answers need current or source-grounded information.
- Consider fine-tuning for repeated behavior or format that can be represented in examples.
- Compare methods on the same held-out tasks and include failure cases.
Return to the Using, adapting, and fine-tuning pretrained LLMs stage · Suggested timing is flexible; move at your own pace.