Lesson 36 of 38 · Written learning
Cost, reliability, and incident response
Estimate resource use and plan how the service will recover when dependencies fail.
A request's cost may depend on input tokens, generated tokens, model choice, and infrastructure time. Estimate with representative usage, then compare the estimate with actual service bills or hardware measurements. Token budgets, maximum output sizes, and usage limits can prevent accidental runaway requests.
Plan for change and failure
Version models, prompts, retrieval data, and application code so a change can be traced. A rollback returns to a known version. Load tests reveal bottlenecks before demand grows. Reliability targets should reflect user needs rather than a copied industry number. An incident process identifies who responds, how to limit harm, and what gets reviewed afterward.
- Set quotas and alert thresholds before opening a service widely.
- Test provider timeouts, malformed output, and missing retrieval evidence.
- Keep a rollback path for model or prompt changes.
- Review incidents for root causes and update the system, not only the prompt.
Return to the Deployment, inference, and optimization stage · Suggested timing is flexible; move at your own pace.