Lesson 32 of 38 · Written learning
Threat-model LLM applications
Recognize prompt injection, information leakage, unsafe output, and excessive tool permissions.
Prompt injection tries to make a model treat untrusted text as controlling instructions. It can arrive from a user, a retrieved document, or a tool response. Data leakage happens when sensitive information is revealed or retained in an unintended place. Insecure output handling can turn generated text into executable code, unsafe HTML, or an invalid database operation.
Reduce what a failure can do
Threat modeling asks what assets matter, who might attack, where untrusted input enters, and what controls limit harm. Apply least privilege to tools, validate output, isolate secrets, and require approval for high-impact actions. Red-team tests should try realistic adversarial inputs and record how the application responds.
- Keep untrusted content clearly separated from developer instructions.
- Sanitize output before rendering it as HTML or using it in another system.
- Limit tool permissions, data access, and action budgets.
- Test prompt injection in retrieved documents and tool outputs, not just direct user prompts.
Return to the Evaluation, security, and responsible AI stage · Suggested timing is flexible; move at your own pace.