Lesson 12 of 38 · Written learning
From text to tokens and embeddings
See how written strings become IDs and vectors that a neural model can process.
Text can contain Unicode characters, punctuation, whitespace, and many languages. Normalization may standardize some of these choices, but changing text carelessly can remove meaningful distinctions. A vocabulary maps common units to token IDs. Modern subword tokenizers can represent frequent sequences efficiently while still handling unfamiliar words as smaller pieces.
IDs are not meaning
A token ID is a label, not a numerical measure of semantic distance. The model looks up an embedding vector for each token, then uses neural layers to build context-sensitive representations. Two similar IDs do not imply two similar meanings; their learned vectors and surrounding context matter.
- Tokenization affects sequence length, costs, and what strings can be represented.
- Different models can tokenize the same sentence differently.
- A special token may mark a boundary, a role, or a task; the exact meaning depends on the model's format.
- When debugging a prompt, inspect the tokenized form rather than assuming word boundaries are preserved.
Return to the NLP and language-model fundamentals stage · Suggested timing is flexible; move at your own pace.