Transformer course: learning-map

Hosted reading edition: download the linked Markdown checkpoint for local notes. Nothing you edit is saved back to this shared website.

Start here every time you return. This is the route through the tutorial, not another prerequisite to read. Your goal is to explain a transformer from the inside by building a tiny model, then adapting a small Qwen checkpoint.

Returning? Open your checkpoint, find the first session whose pass gate you have not demonstrated, then open its numbered stop below. There is no automatic progress tracking: update progress.md in Otty. The browser pages are reading copies.

See the whole journey

Open the animated eight-stage journey. The diagram shows the recommended learning order, not training execution or measured progress. Use Live/Still to trace or pause it. The numbered stops below are the actionable navigation.

Stage	Open your next stop	What you can explain afterward
1. Text enters the model	01: the map, 02: tokens	Text, IDs, roles and special tokens; tokenizer versus neural decoder.
2. Vectors become context	03: embeddings, 04: attention	Shapes, vocabulary scores, Q/K/V, causal masks and position.
3. Blocks and inference	05: a full block, 06: cache	Residuals, norms and MLPs; prefill, decode and temporary state.
4. Learn from examples	07: one update, 08: tiny pretraining	Targets, loss, gradients, optimizer updates and saved weights.
5. Define specialization	09: post-training, 10: baseline	SFT versus other adaptation stages; honest data splits and baselines.
6. Adapt locally	11: LoRA and masks, 12: smoke run	Frozen versus trainable weights; the exact supervised completion.
7. Improve and debug	13: controlled tuning, 14: errors	Which change caused a gain, and which failures remain.
8. Measure and explain	15: speed, 16: teach-back	Quality, latency and memory evidence; independently reproducible results.

Plan two sessions per week, with 10-12 calendar weeks available for repetition. Later sessions may need several visits. The pass gate, not elapsed time, decides when you move on.

How to use each stop

Readiness boundary: Session 1 is a complete local worksheet. Sessions 2-16 below provide materials, tasks and gates, but their toy code, notebooks, dataset builder and task evaluator are not prebuilt local labs. You will construct them using the linked original code and documentation. The Mac command sheet contains source-checked examples, not an installed, pinned or executed environment. A smoke run is not a benchmark, and this is not an exact reproduction of Jev.

01 · Trace the whole loop

Prerequisite: none. Tools: browser; your existing Python for the byte exercise. No packages.

Do: follow the timed Session 1 worksheet: sort parts, phases and state; round-trip byte IDs; narrate one generated label; compare prefill/decode; give the two-minute teach-back.

Pass: explain text → IDs → embeddings → blocks → scores → selected ID → text. Do not confuse tokenizer encoding with an encoder stack, or cache with learned weights. Save six sentences or a short recording.

If stuck: repeat the worksheet section for the confusing term; do not install Qwen to compensate for a missing mental model.

02 · Tokens and chat templates

Prerequisite: 01. Setup gate: follow the separate Mac environment route. Verify arm64 Python and package compatibility before installing anything. Tokenizer inspection needs checkpoint tokenizer files, not model weights.

Do: use the real Qwen tokenizer on room, Room, room, multilingual text and the same sentence in a chat template. Save rendered text, IDs, special tokens, round trips and the tokenizer revision. Use minbpe for the teaching exercise, never as a replacement tokenizer for Qwen.

Pass: show which IDs belong to formatting versus content; explain why a character, word and token need not coincide.

If stuck: repeat Session 1’s byte round trip, then inspect one plain sentence before adding roles. A missing package is a setup issue, not a model concept; use the current upstream instructions.

03 · IDs become vectors, vectors become scores

Do: locate the embedding lookup and vocabulary head. Write the shapes for an input of one batch and three positions; inspect two lookups of the same ID. Change the selected ID and compare its embedding. Record hidden width and vocabulary size from the actual model configuration.

Pass: distinguish integer IDs, hidden vectors and vocabulary-wide scores. The initial lookup of the same ID is the same; contextual representations can differ after blocks. Check whether head and embeddings share weights.

If stuck: use the embedding portion of Karpathy’s MLP lesson, then draw one lookup instead of reading the entire implementation.

04 · Who may read which token?

Do: draw the allowed reads for three causal positions. Locate Q/K/V and the mask in the teaching implementation. Change a future token and compare an earlier position in evaluation mode with dropout disabled. Then inspect Qwen’s query/KV head counts and RoPE code.

Pass: earlier causal positions cannot read future tokens; distinguish Q, K and V, and explain that GQA can have fewer KV heads than query heads. Attention pictures are not guaranteed causal explanations of a decision.

If stuck: reduce the example to three tokens and label permitted reads before considering multiple heads. RoPE details can wait; identify its role.

05 · Reconstruct one complete block

Do: trace norm → attention → residual addition → norm → MLP → residual addition. Mark both saved bypasses. Compare an encoder’s bidirectional mask with the causal decoder. Locate Qwen’s RMSNorm/RoPE rather than assuming nanoGPT’s LayerNorm/learned positions are identical.

Pass: redraw the block without notes; explain that attention mixes token information while the MLP transforms features per position. GPT/Qwen does not need a separate encoder stack.

If stuck: trace one residual addition separately, then return to the whole block. Do not treat the block as a replacement-only pipeline.

06 · Separate persistent weights from request state

Do: start with the worksheet’s token-position counts. For an executable small model, compare short greedy generation with and without KV caching under identical settings. Record IDs, prompt/output counts and per-layer cache shapes. Use a reference that supports both paths; do not simulate a cache by merely reusing the output text.

Pass: output IDs agree under matched numerics/settings, or any numerical divergence is investigated. Explain first-token prefill, subsequent decode and cache growth. No model parameters change during generation.

If stuck: return to the 128-prompt/16-output count exercise. Learn the mechanism before measuring speed; a count ratio is not a timing prediction.

07 · Observe one update, not a mysterious training run

Do: reproduce a small original-author example. Trace prediction, target, loss, gradient and optimizer update. Save a parameter before/after the update; keep another parameter frozen. Deliberately overfit one tiny batch.

Pass: an intended trainable value changes, a frozen value does not, and you can explain why the shifted next-token target differs from generation. You need shapes and an error-reduction story, not calculus derivations.

If stuck: follow one scalar in micrograd before returning to tensors. Check that gradients exist and the optimizer actually steps.

08 · Train and reload a glass-box toy

Do: build the course’s small two-block, byte-tokenized toy using permitted synthetic text. Inspect one input batch and its one-position-shifted targets. Start on CPU with short context and a few minutes of training. Save weights and configuration; reload them in a fresh process and generate greedily.

Pass: reload reproduces output under the saved settings. Report falling training loss as toy learning or memorization, not useful domain accuracy. This local toy is to be built; it is not already supplied by the tutorial.

If stuck: overfit one batch again before enlarging the corpus or model. Use the compact code as an educational reference, not a Mac CUDA run script.

09 · Locate pretraining and post-training in the lifecycle

Do: annotate which stages change weights: pretraining, optional continued pretraining, SFT, preference/RL methods and distillation. Contrast these with prompting and inference. Compare genuinely matched base/instruct models if available; otherwise inspect their documentation and compare prompting modes without pretending this isolates post-training.

Pass: explain why learning a label from examples is SFT, while requesting a short answer or disabling visible thinking at inference is not training. Name which methods this capstone will omit.

If stuck: sort the lifecycle into “changes parameters” and “uses existing parameters”; do not try RL before understanding supervised learning.

10 · Make an experiment that can be wrong

Do: write the BUG/FEATURE/UNCLEAR rules and ambiguous examples. Group permitted examples by source/template family before train/dev/test splitting; check duplicates and labels independently. Freeze majority, simple text-classifier and prompted-Qwen baseline predictions/configurations. The twelve synthetic examples only test formatting, not model quality.

Pass: a saved manifest proves split separation; you can compute per-class counts and explain the baseline. Choose numeric success thresholds after the baseline and before tuning, not after seeing adapter results.

If stuck: resolve disputed labels and leakage before increasing data. A poorly specified label is not fixed by a larger model.

11 · See exactly what LoRA learns

Do: serialize one chat training example and inspect its IDs and actual loader loss mask. Confirm the label and ending token survive truncation. Identify adapter target modules and count trainable/frozen parameters. Inspect the empty thinking block: the inference flag is not automatically forwarded to the training loader.

Pass: point to every supervised completion token, explain which prompt tokens are masked, and show which parameters an optimizer will update. An option named --mask-prompt alone is not evidence.

If stuck: inspect a single final-assistant example before batching or training. Do not strip checkpoint template tokens to make a diagram simpler.

12 · Prove that training, saving and reloading connect

Prerequisite: 11, including the inspected mask. Watch/inspect: the conservative Mac smoke command and current installed CLI help.

Do: run the bounded 20-update training example on train data with dev validation, one model process, batch one and short sequences. Observe system memory pressure. Save the adapter, reload it in a fresh process and repeat a fixed dev prediction under the same generation settings.

Pass: local save/reload works, memory stays within your measured budget, and the test set has not influenced selection. Save command, package/model revisions, mask evidence and adapter identity. No accuracy claim yet.

If stuck: reduce context/batch and investigate the explicit failure. Do not launch a second model or extend training before reload works.

13 · Change one factor at a time

Do: choose one change: examples, rank, learning rate or update count. Keep the remaining configuration fixed; record the dev result and compare with the frozen baseline. Use dev quality to select and stop; do not peek at test outcomes to choose a checkpoint.

Pass: save one reproducible candidate and an honest comparison, including a failure to improve. Explain what the controlled comparison can attribute to training and what it cannot.

If stuck: inspect dev mistakes in 14 before changing several knobs. No improvement is an informative result, not permission to hide the baseline.

14 · Read the mistakes

Prerequisite: 13; use dev data. Watch/inspect: the course’s model report card.

Do: implement or use an explicit task evaluator for confusion counts, per-class precision/recall, macro-F1 and invalid labels. Review mixed, ambiguous, paraphrased and out-of-domain dev examples. Pair baseline/candidate errors; label remaining failure types rather than presenting only successes.

Pass: every gain has counts and examples, and invalid outputs remain explicit errors rather than silently becoming UNCLEAR. Loss/perplexity from the training CLI is not the task evaluator. The evaluator is a lab artifact you still need to construct.

If stuck: manually check a small confusion matrix first. Return to 10 if the label rubric itself is ambiguous; use only dev findings for revisions.

15 · Measure short decisions honestly

Do: make the final quality comparison on the frozen holdout. Separately collect matched baseline/adapter timing samples: same hardware, prompts, output limits, template, precision and cache policy. Label warm versus cold runs; record sample counts, first-token time, end-to-end p50/p95, token counts and measured memory pressure. Quantization is a separate comparison.

Pass: report quality together with speed and memory. Do not equate smaller weight files or shorter prose with an isolated adapter-training speedup. If you revise after seeing test results, use a new untouched holdout.

If stuck: begin with one instrumented request, then repeated samples. Do not report percentiles from one run or call model loss classification F1.

16 · Finish with evidence, not confidence

Prerequisite: 15. Watch/inspect: all three component/lifecycle maps, your saved artifacts and the actual checkpoint configuration.

Do: draw a prompt-to-text loop and a decoder block from memory. Locate each component in code; explain one training update, one inference request and the base/adapter difference. Give another person the frozen configuration, data manifest, adapter identity and evaluator so they can reproduce the run without this chat.

Pass: a fresh process reproduces the saved comparison, and an independent reader can locate the components and verify the reported counts. Explain why this teaches a System 1-like behavior without reproducing Jev’s architecture. Keep the learning and model-quality report cards separate.

If stuck: follow the missing evidence back to its specific stop: tokens 02, blocks 05, state 06, updates 07, masking 11 or measurement 15.

Save your place and return without asking this chat

Edit progress.md in Otty: change only sessions whose gates you actually passed, and fill in the Resume here block. Do not mark a learner session complete because an assistant generated a page or ran a command.

pandoc progress.md --from markdown --to html5 --standalone \
  --toc --toc-depth=2 --css=course.css \
  --metadata pagetitle="Course checkpoint" --output progress.html

Your next visit starts at this map, then your checkpoint, then the first unpassed gate. All learning notes and experimental outputs stay in the dedicated study folder; large/generated runs and models remain ignored.

Your transformer learning map