Lesson 19 of 38 · Written learning
Data provenance, licenses, and quality
Treat the dataset as an engineered and documented part of the model, not a pile of text.
Before collecting text, define the permitted source and intended use. Record the source, license or permission, collection date, transformations, and known limitations. Deduplication can reduce repeated documents; filtering can remove malformed or irrelevant material. Each filter can also remove useful language, so keep a record of what it changes.
Privacy and evaluation contamination
Training data may contain personal or sensitive information. Do not assume public availability means unrestricted reuse. Benchmark contamination occurs when test material or close copies appear in training data; it can make an evaluation look better than a model's ability to generalize. Separate train, validation, and test data before repeated experimentation.
- Prefer authorized, documented, and purpose-appropriate data.
- Keep a dataset card or concise provenance note.
- Do not place personal information in a student corpus without a lawful and clearly justified basis.
- Use hashes or similarity checks to discover duplicates, then inspect a sample of both kept and removed records.
Return to the Data engineering and pretraining stage · Suggested timing is flexible; move at your own pace.