Start here
From first principles to a working system.
Build, evaluate, document, and deploy an LLM-powered application while understanding its underlying ideas and trade-offs.
- Prerequisites
- Basic computer literacy. Programming and mathematics are introduced from the ground up.
- Tools
- Python, NumPy, PyTorch, Hugging Face, Jupyter, Git, Optional JavaScript / TypeScript for integration.
- Expanded pace
- Approximately 36–48 weeks, at your own pace.
The learning route
Four levels. Thirteen stages.
Follow the sequence if you are new to the field, or jump to a subject you already know. Suggested timings are planning estimates, not deadlines.
Foundations
Python, mathematics, machine learning, deep learning, and NLP. Build the knowledge needed to understand LLMs rather than treating them as black boxes.
STAGE 00Environment and prerequisites
Get oriented to AI and prepare a small, reproducible workspace before learning model internals.
Topics in this stage
- How AI, machine learning, deep learning, generative AI, and LLMs relate
- Python, virtual environments, notebooks, Git, and GitHub
- CPU or no-cost cloud notebook options
- Documentation, packages, command-line basics, and debugging
- A map of AI, machine learning, and LLMsPlace language models inside the wider set of ideas called artificial intelligence.
- Set up a reproducible learning workspacePrepare Python, notebooks, Git, and a repeatable place to run experiments.
Practical work · ungraded
Create a Git repository, run a Python notebook, load a text file, and record the environment you used.
By the end
A working development setup and a reproducible first notebook.
STAGE 01Python and software foundations
Learn enough programming to inspect data, write small utilities, and understand the examples used later.
Topics in this stage
- Variables, types, collections, conditions, loops, functions, classes, and exceptions
- CSV, JSON, and text files
- NumPy arrays and vectorized operations
- HTTP, APIs, JSON responses, environment variables, secrets, Git, dependencies, and debugging
- Python data and control flowUse variables, collections, conditions, and loops to express a small program.
- Functions, files, and errorsOrganize small programs and read common text-data formats safely.
- Arrays, APIs, and version controlConnect numerical data, web requests, and software history.
Practical work · ungraded
Build a text utility that cleans documents, counts words, extracts metadata, and exports JSON.
By the end
Ability to write and debug small Python programs and follow later machine-learning code.
STAGE 02Mathematics for LLMs
Build intuition for the numerical operations used in embeddings, attention, probability, and training.
Topics in this stage
- Scalars, vectors, matrices, tensors, shapes, and matrix multiplication
- Dot products, transposes, norms, and cosine similarity
- Functions, derivatives, partial derivatives, and the chain rule
- Probability, expectation, entropy, sampling, statistics, logarithms, exponentials, softmax, cross-entropy, gradient descent, and backpropagation
- Vectors, matrices, tensors, and shapeRead the basic language of numerical arrays and understand why shape matters.
- Derivatives, gradients, and learningSee how a model can use an error signal to improve its parameters.
- Probability, logits, and lossConnect model scores to probabilities and to the objective used during training.
Practical work · ungraded
Use NumPy to implement cosine similarity, softmax, cross-entropy, and a small gradient-descent example.
By the end
Ability to explain the mathematical operations used in neural networks and transformers.
STAGE 03Machine learning and deep learning
Understand what training data tells a model, how neural networks transform inputs, and how training is diagnosed.
Topics in this stage
- Supervised, unsupervised, and self-supervised learning
- Training, validation, and test splits
- Features, labels, loss, optimization, and regularization
- Overfitting, underfitting, generalization, and data leakage
- Neural networks, layers, activations, parameters, and embeddings
- Forward/backpropagation, optimizers, learning rates, tensors, batching, automatic differentiation, and GPU fundamentals with PyTorch
- How machine learning learns from examplesDistinguish learning setups and design data splits that make evaluation more honest.
- Neural networks and representationsFollow how learned layers transform numbers into useful internal representations.
- Training a small model with PyTorchConnect tensors, automatic differentiation, batches, and optimizers in a training loop.
Practical work · ungraded
Train a small neural network and compare its behavior on training and validation examples.
By the end
Understanding of parameter learning and basic training diagnostics.
STAGE 04NLP and language-model fundamentals
Move from raw text to token sequences, then understand language modeling as prediction over those sequences.
Topics in this stage
- The evolution from rule-based NLP to neural language models
- Text normalization, vocabulary, tokens, and subword tokenizers
- Word embeddings and contextual representations
- N-gram models and next-token prediction
- Sequence modeling and long-range context
- Training objectives, logits, probabilities, perplexity, classification, and generation
- From text to tokens and embeddingsSee how written strings become IDs and vectors that a neural model can process.
- Language models and next-token predictionUnderstand how simple n-gram models lead to the central training objective of an LLM.
- Logits, probabilities, and perplexityInterpret model outputs without confusing a score with a truth judgment.
Practical work · ungraded
Build a character-level language model, generate text, and compare its output with an n-gram baseline.
By the end
Ability to explain how a language model learns patterns and predicts the next token.
Understanding LLMs
Study tokenization, transformer architecture, pretraining, inference, and adaptation so models become understandable systems rather than black boxes.
STAGE 05Transformers and LLM architecture
Understand attention, transformer blocks, common architecture families, and how an autoregressive model produces a response.
Topics in this stage
- Attention and earlier sequence-model limitations
- Query, key, and value projections
- Scaled dot-product attention and causal masks
- Multi-head attention and positional information
- Feed-forward blocks, residual connections, and normalization
- Decoder-only, encoder-only, and encoder–decoder models
- Parameters, context windows, inference, and decoding strategies
- Why attention helps language modelsConnect the idea of context to the attention mechanism used in transformers.
- Scaled attention, masks, and multiple headsSee why attention scores are scaled and how masking controls which positions a token can use.
- Transformer blocks and model familiesAssemble attention with feed-forward layers and compare encoder, decoder, and encoder–decoder designs.
- Inference and decoding strategiesUnderstand how token probabilities become a visible response.
Practical work · ungraded
Implement scaled dot-product attention, visualize a causal mask, and assemble a small decoder-only transformer.
By the end
Ability to trace how token representations become contextual and how a transformer predicts a next token.
Major project: Build a tiny GPT-style language model and train it on a small text corpus whose license permits the intended use.
STAGE 06Data engineering and pretraining
Follow data from its source to a training stream and understand the small-scale pretraining loop and its limits.
Topics in this stage
- Collecting and licensing data
- Deduplication, filtering, normalization, and quality checks
- Personal data, provenance, contamination, and benchmark leakage
- Tokenization pipelines, data loading, and next-token objectives
- Loss curves, checkpoints, schedules, clipping, compute, memory, throughput, failures, and scaling laws
- Data provenance, licenses, and qualityTreat the dataset as an engineered and documented part of the model, not a pile of text.
- Build a tokenization and data pipelineTurn prepared documents into repeatable sequences that a training loop can consume.
- Pretraining, checkpoints, and computeUnderstand the next-token training loop and why student projects use small models.
Practical work · ungraded
Prepare a documented corpus, train a small model, track loss, and compare the effect of data-cleaning choices.
By the end
Understanding of the data pipeline and the trade-offs involved in pretraining.
Major project: Build a reproducible small-scale pretraining pipeline. Frontier-scale model training is outside the required student budget.
STAGE 07Using, adapting, and fine-tuning pretrained LLMs
Choose an appropriate model, use its prompt format correctly, and compare prompting, retrieval, and fine-tuning.
Topics in this stage
- Model families, model cards, open weights, and hosted models
- Tokenizers, chat templates, system instructions, roles, batching, and context limits
- Instruction tuning and supervised fine-tuning
- Dataset quality, full fine-tuning, LoRA, QLoRA, quantization, adapters, and checkpoints
- Preference optimization, DPO, RLHF, and choosing prompting versus retrieval versus tuning
- Choose and run a pretrained modelCompare hosted services and open-weight models while accounting for licenses and operating limits.
- Instruction tuning with examplesUnderstand how supervised fine-tuning teaches a pretrained model to imitate desired responses.
- LoRA, quantization, and preference tuningCompare adaptation methods and understand what preference optimization is trying to change.
Practical work · ungraded
Adapt a small compatible pretrained model on a curated instruction dataset and compare it with its base model.
By the end
Ability to choose an adaptation strategy and measure whether it improves the intended behavior.
Building LLM systems
Turn model capabilities into useful software with retrieval, tools, evaluation, security, and responsible deployment.
STAGE 08Embeddings, vector search, and RAG
Ground model responses in selected documents using retrieval, then test whether evidence actually supports an answer.
Topics in this stage
- Semantic embeddings and similarity search
- Parsing, chunking, overlap, and metadata
- Vector indexes, keyword search, and hybrid retrieval
- Query rewriting, filtering, reranking, and retrieval evaluation
- RAG pipelines, citations, missing evidence, and prompt-injection risks
- Local and managed vector storage
- Embeddings and similarity searchUnderstand how vector representations support semantic search and where similarity can mislead.
- Build a retrieval-augmented generation pipelineTrace a question from document ingestion through retrieval and answer generation.
- Evaluate retrieval, evidence, and citationsFind out whether a RAG system retrieves the right passages and stays within what they support.
Practical work · ungraded
Build a question-answering application over controlled documents; cite source passages and admit when evidence is insufficient.
By the end
Ability to build and diagnose a basic retrieval-augmented generation system.
Major project: Create a document research assistant with retrieval, source citations, evaluation cases, and a clear privacy policy.
STAGE 09LLM application engineering and agents
Wrap model calls in ordinary software boundaries, validate outputs, and choose tools and workflows with appropriate safeguards.
Topics in this stage
- Application layers and provider abstractions
- Structured outputs, JSON schemas, and validation
- Function calling and tools
- Multi-step workflows, state, agents, memory, planning, retries, timeouts, and stopping conditions
- Human approval for consequential actions
- Untrusted tool output, permissions, web applications, and APIs
- Structured outputs and tool callsUse schemas and validation to turn free-form model output into safer application data.
- Workflows, agents, and human approvalChoose between a predictable workflow and a model-driven loop.
- Integrate an LLM into a web applicationPlace model calls behind a clear software boundary with dependable errors and provider choices.
Practical work · ungraded
Build an assistant that calls a read-only tool, validates returned data, and asks for human approval before consequential actions.
By the end
Ability to design around an LLM without treating it as a fully reliable decision-maker.
STAGE 10Evaluation, security, and responsible AI
Test real tasks, threat-model the full system, and document risks and limitations before release.
Topics in this stage
- Unit, integration, and regression testing
- Task-specific metrics and human evaluation
- LLM-as-judge limits and evaluator calibration
- Hallucination, factuality, grounding, and abstention
- Prompt injection, data leakage, unsafe output, and tool security
- Privacy, consent, copyright, bias, accessibility, red teaming, documentation, reproducibility, and release criteria
- Evaluate LLM systems with task-specific testsBuild a test set that reflects what users will actually ask and how failures matter.
- Threat-model LLM applicationsRecognize prompt injection, information leakage, unsafe output, and excessive tool permissions.
- Privacy, bias, and responsible releaseDocument what the system can do, what it cannot do, and what risks remain.
Practical work · ungraded
Create normal, edge-case, adversarial, and insufficient-evidence prompts; track results across system changes.
By the end
Ability to describe limitations and decide whether a system is suitable for a limited release.
STAGE 11Deployment, inference, and optimization
Operate a small LLM service with sensible limits, observability, cost awareness, and safe failure behavior.
Topics in this stage
- Serving models through APIs and application backends
- Local and hosted inference
- GPU memory, quantization, batching, caching, and throughput
- Latency, concurrency, streaming, and rate limits
- Deployment environments, secrets, logging, and observability
- Cost estimation, tokens, retries, fallbacks, versioning, rollback, monitoring, incident response, data retention, and load testing
- Serve models and improve inferenceBalance latency, throughput, memory, and quality when a model answers real requests.
- Deploy safely and observe the systemMove from a local prototype to a service that handles failures and protects secrets.
- Cost, reliability, and incident responseEstimate resource use and plan how the service will recover when dependencies fail.
Practical work · ungraded
Deploy an LLM application with error handling, evaluation checks, usage limits, and basic monitoring.
By the end
Ability to operate a small LLM service responsibly and explain performance and cost trade-offs.
Applied engineering and capstone
Combine the course ideas in an end-to-end project, document design decisions, and select a research or systems direction for continued learning.
STAGE 12Capstone and advanced directions
Bring the engineering ideas together in a documented application, then choose an advanced direction for further learning.
Topics in this stage
- Translate a real problem into a testable product specification
- Choose a model, data, architecture, and evaluation strategy
- Build and document a complete application
- Measure quality, latency, reliability, and cost
- Explain limitations and design decisions
- Explore multimodal models, long context, distillation, mixture-of-experts, distributed training, or research-paper reproduction
- Plan an end-to-end LLM capstoneTurn a meaningful problem into a bounded system you can implement and evaluate.
- Choose an advanced directionSurvey several research and engineering areas and choose one to explore with a focused reading plan.
Practical work · ungraded
Deliver a tested, documented LLM application and present what it does, how it behaves, and where it fails.
By the end
A portfolio project that demonstrates end-to-end engineering judgment—not a guarantee of mastery or employment.
Build as you learn
Projects that connect the ideas.
These are practical learning activities, not graded assessments. Scale the tools and datasets to your available hardware.
Build a tiny language model
Trace the path from a permitted text corpus to a small next-token model.
- Prepare and document a small licensed text dataset.
- Implement tokenization and a basic training loop.
- Train a character-level baseline and a small transformer.
- Plot loss, inspect generations, and explain where the model fails.
Possible deliverable: Source code, dataset notes, training curves, and an experiment report.
Fine-tune a pretrained model
Adapt a compatible small model and compare its behavior with the base model.
- Choose a model with a license that permits the intended use.
- Curate an instruction dataset and hold out comparison examples.
- Fine-tune with an appropriate method such as LoRA.
- Record compute requirements, limitations, and failure cases.
Possible deliverable: A reproducible training script, model configuration, comparison results, and a model card.
Build a citation-based RAG assistant
Answer questions over controlled documents while keeping evidence visible.
- Ingest public or authorized documents.
- Create chunks, metadata, and embeddings.
- Retrieve and rank relevant passages.
- Generate grounded answers, cite sources, and test unsupported questions.
Possible deliverable: A working application, source references, evaluation cases, and a retrieval-quality report.
Deploy an LLM application
Move a prototype into a limited service with explicit safeguards and useful monitoring.
- Provide a documented API and usable interface.
- Add input validation, timeouts, retries, and usage limits.
- Measure latency, token use, and approximate cost.
- Add regression and security tests, monitoring, and safe failure behavior.
Possible deliverable: A deployed or locally runnable application, architecture diagram, test notes, and operating guide.
Final capstone: an end-to-end LLM product
Choose a meaningful use case, select an architecture, implement it, document it, and measure its behavior.
- Define the user, task, boundaries, and success criteria.
- Select the model, data, architecture, and evaluation strategy.
- Build the system and test normal, edge, adversarial, and unsupported cases.
- Measure quality, latency, reliability, and cost; explain limitations.
Possible deliverable: A project repository, demo, technical report, architecture notes, and a short presentation of findings.
Go deeper
References and further reading.
These sources support the lessons; they are not a substitute for the written explanations. Documentation and APIs can change, so check current versions.
Transformer architecture
Attention Is All You Need
Vaswani et al. (2017), the paper introducing the Transformer architecture.
NLP and pretrained models
Hugging Face LLM Course
A companion reference for tokenizers, datasets, Transformers, and the Hugging Face ecosystem.
Language models from scratch
Stanford CS336: Language Modeling from Scratch
Advanced material on data, Transformer construction, training systems, and evaluation; stronger prerequisites are expected.
Deep-learning implementation
PyTorch Tutorials
Official tutorials for tensors, neural networks, automatic differentiation, optimization, and model saving.
Model training
Transformers training documentation
Current Hugging Face documentation for training and adapting Transformer models.
Retrieval-augmented generation
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis et al. (2020), an early foundational paper on combining retrieval with generation.
Post-training
Hugging Face TRL
Reference documentation for supervised fine-tuning, preference methods, reward modeling, and related techniques.
Application security
OWASP Top 10 for LLM and Generative AI Applications
Security guidance on risks such as prompt injection, sensitive-information disclosure, and excessive agency; check the current edition.
Evaluation
Evaluation resources
An example of evaluation tooling and practices; vendor-specific tooling is one option, not the only method.
Neural-network foundations
Deep Learning
The online textbook by Goodfellow, Bengio, and Courville.
Machine-learning foundations
scikit-learn User Guide
A practical reference for common machine-learning methods and evaluation concepts.
Datasets
Hugging Face Datasets documentation
Documentation for loading, processing, and sharing datasets in the Hugging Face ecosystem.
Training systems
Stanford CS336 lecture materials
Lecture materials for a deeper study of language-model systems and training.
Responsible AI
NIST AI Risk Management Framework
A framework for identifying and managing AI-related risks.
Model serving
vLLM documentation
Documentation for a high-throughput LLM inference and serving engine.
Practical activities and projects are ungraded learning work. This course has no quizzes, scores, or formal assessments. Hardware and hosted-service availability can change; use small, permitted datasets and check current terms before relying on a service.