Lesson 31 of 38 · Written learning
Evaluate LLM systems with task-specific tests
Build a test set that reflects what users will actually ask and how failures matter.
Begin by defining a task and the observable qualities that matter. Exact match can work for a narrow structured task but fails to capture equivalent wording. Human review can judge usefulness or tone, but reviewers need a clear rubric and representative examples. Automated metrics are repeatable, yet measure only what their design captures.
Treat an LLM judge as another model
An LLM-based evaluator can help compare responses at scale, but it may favor longer answers, share biases with the model being evaluated, or miss subtle errors. Calibrate it against human-reviewed examples and report disagreement. Keep a small, stable regression set and separate it from examples used repeatedly to tune prompts.
- Include common requests, edge cases, ambiguous inputs, and unsupported questions.
- Record the system version, prompt, model, and evaluation method.
- Review failures as categories rather than reporting only an average.
- Test retrieval and tool behavior in addition to the final wording.
Return to the Evaluation, security, and responsible AI stage · Suggested timing is flexible; move at your own pace.