61 questions you should expect, with answers. Every question here is one an audience actually asks during Transformers from First Principles (Lecture 01, 22 slides). They're grouped by topic rather than by slide, so you can find one fast mid-Q&A — the slide reference sits under each answer if you want to jump back to the visual.
Companion to: the deck and the takeaway repo
- Prerequisites and framing
- Why this architecture matters
- The problem, and the RNN era
- Attention: the idea
- Attention: the mechanics
- Position
- Inside the block
- The three architecture flavors
- Training
- Tokenization
- Speed, cost and scaling
- What's actually in production
- The mental model
- The takeaway repo, and what next
Matrix multiply and dot product — that's the whole list. If you can multiply two matrices you can follow every slide. Anything heavier gets narrated rather than derived. There is exactly one equation in the lecture and it's four operations in a row.
Slide 1 — Title & frame
Same architecture, different scale. The repo you leave with is a decoder-only transformer — structurally the same family as GPT-4, just thousands of times smaller and trained on Shakespeare instead of the internet. The block diagram on slide 12 is the block diagram in a frontier model; the difference is how many of them are stacked and how much text went through.
Slide 1 — Title & frame
Not to call an API. But it explains context limits, token pricing, latency behaviour, and why prompts fail the way they do — and that's most of the debugging you'll actually do. The people who get stuck are usually the ones treating the model as an oracle rather than a next-token predictor with a fixed context window.
Slide 22 — Q&A
The specific optimisations churn yearly — Flash Attention, GQA, MoE all arrived after 2020. But the five-step mental model has held since 2017. Learn the model; treat the optimisations as news.
Slide 22 — Q&A
Type out the repo by hand rather than cloning it, and break something on purpose every time you understand a piece. Reading this architecture is easy; implementing it is where it sticks. Delete the causal mask and watch the loss collapse — then work out why.
Slide 22 — Q&A
No — they're still better when data is scarce, because convolution bakes in an assumption about locality that a transformer has to learn from examples. Vision Transformers win at scale; CNNs still win on small datasets and tight edge budgets. What ended was the assumption that vision requires convolution.
Slide 2 — Why bother
Different training objective, often the same backbone. Modern image models increasingly put a transformer inside the denoiser (the DiT family). The block drawn in this lecture shows up there too — what changes is what you're predicting, not what's doing the predicting.
Slide 2 — Why bother
State-space models like Mamba are the live challenger, and hybrid architectures are shipping. But nothing has displaced attention at frontier scale yet. The concepts transfer either way — sequence modelling, parallel training, next-token objectives all survive whatever wins.
Slide 2 — Why bother
Because it's the only objective with unlimited free training data, and it turns out that doing it well requires modelling grammar, facts and reasoning. A cleverer objective that needs human labels can't reach the same scale, and scale is what made the difference.
Slide 3 — Sequence modeling
That's failure mode three on the slide — a bag of words throws away order, so "dog bites man" and "man bites dog" become the same input. Worth noting: attention on its own has exactly this bug. Slide 11 is where it gets fixed.
Slide 3 — Sequence modeling
"What exactly is a hidden state?"
A fixed-size vector — say 512 numbers — that gets overwritten at every timestep. That fixed size is the problem: a whole novel has to be compressed into the same 512 numbers as a three-word sentence, and there's no mechanism to decide the novel deserves more room.
Slide 4 — RNNs & LSTMs
Rarely for language, but they're alive in low-latency streaming and tiny-device settings where constant memory per step beats attention's quadratic cost. The state-space revival (Mamba, S4) is a modernised descendant of the same idea.
Slide 4 — RNNs & LSTMs
They soften it. A gate gives information a protected path so it decays more slowly — but it's still a path through every intermediate step, and a thousand steps of even gentle decay is still decay. Attention removes the path entirely: token 1 is reachable from token 900 in one hop.
Slide 5 — Two fatal flaws
Generation is sequential either way — RNN or transformer — because you need token t before you can produce t+1. The win described in this lecture is on training, where the whole sequence is known up front and can be processed in one parallel pass.
Slide 5 — Two fatal flaws
Backpropagating through a thousand steps multiplies a thousand numbers together. If they're slightly below one, the product goes to zero; slightly above, it explodes. It's repeated multiplication, nothing more exotic than that. Slide 13's residual connections are the fix for depth; attention is the fix for sequence length.
Slide 5 — Two fatal flaws
No — attention existed since 2014 (Bahdanau et al.) as a bolt-on to RNN-based translation models. The 2017 contribution was noticing that the attention was doing the work and the recurrence was just getting in the way, then removing everything else.
Slide 6 — Attention Is All You Need
Yes — this is the architecture's one real weakness. It's O(n²) in sequence length, in both compute and memory. We pay it because it parallelises, which turns out to be worth far more. Slides 17 and 19 both come back to the bill.
Slide 6 — Attention Is All You Need
Three separate learned weight matrices, applied to the same token embedding. Same input vector, three different projections. Nobody hand-designs them — they're learned by gradient descent like everything else in the model.
Slide 7 — Attention, intuitively
Because relevance isn't symmetric and isn't single-purpose. Splitting the roles lets a token say "this is what I'm looking for" separately from "this is what I am" — and lets what it broadcasts differ from what it actually hands over when selected. Comparing raw embeddings collapses all three jobs into one vector and forces relevance to mean similarity, which isn't what language needs.
Slide 7 — Attention, intuitively
Nothing tells it that. It discovers the pattern from data because resolving pronouns correctly lowers next-token loss. This is also why attention maps became the standard interpretability tool — you can look at what the model decided and check whether it matches your intuition.
Slide 7 — Attention, intuitively
Dot products of d-dimensional random vectors grow in magnitude like √d. Feed large numbers into softmax and it saturates — almost all the weight lands on one token and the gradients vanish. Dividing by √d cancels that growth exactly, keeping scores in a range where softmax stays informative. This is a favourite interview question; don't skip it.
Slide 8 — Self-attention, formally
Softmax is differentiable and it's sharpening: exponentiating exaggerates the gap between the best match and the rest. That lets attention be nearly-hard when the evidence is strong and genuinely soft when it isn't. Plain normalisation would keep everything mushy and also breaks on negative scores.
Slide 8 — Self-attention, formally
Self-attention means Q, K and V all come from the same sequence — tokens attending to each other. Cross-attention takes Q from one sequence and K/V from another; that's how the 2017 decoder reads the encoder, and how many multimodal models attach an image encoder to a language model. Same formula either way.
Slide 8 — Self-attention, formally
Because the mask is applied before the softmax. A score of zero is still a competitive score — it would come out of softmax with real weight. exp(-inf) is a true zero, so the row still sums to one across only the legal positions.
Slide 9 — The attention matrix
It's harmless but redundant — at generation time the future doesn't exist yet. The mask matters during training, and that's precisely what lets you train on an entire sequence in parallel while still learning left-to-right prediction.
Slide 9 — The attention matrix
Yes, and you should — it's an afternoon's work with any Hugging Face model, which will hand you attention weights if you ask for them. Attention maps are the most inspectable artefact in the architecture, and interpretability research largely starts here.
Slide 9 — The attention matrix
Practically: aim for a head dimension around 64 to 128, then divide your embedding width by that. GPT-2 small uses 12 heads over 768 dimensions. It's a tuning choice, not a deep truth.
Slide 10 — Multi-head attention
Some do, provably — induction heads are the famous documented case. But many heads are redundant; several papers have pruned a majority of them at little cost to performance. Treat the four labels on the slide as illustrative of what heads can learn, not a taxonomy of what they do learn.
Slide 10 — Multi-head attention
Roughly the same FLOPs, because each head is narrower — you're re-slicing the same budget, not adding to it. The only extra cost is the output projection that recombines the concatenated heads.
Slide 10 — Multi-head attention
It feels wrong and it works. The space is high-dimensional enough that the model learns to separate the two contributions. That said, the discomfort was productive: it nudged the field toward RoPE, which keeps position out of the embedding entirely and applies it inside attention instead.
Slide 11 — Position
Because language cares about distance, not addresses. "The adjective right before the noun" is a relative fact, and it should mean the same thing at position 10 and at position 10,000. Absolute schemes force the model to learn that equivalence separately at every position.
Slide 11 — Position
Part of it — position interpolation stretches RoPE beyond the range it was trained on, which is why context windows could be extended without full retraining. The other half of the problem is making attention itself cheaper, which is slide 19.
Slide 11 — Position
Roughly two-thirds in a standard block. The feed-forward network expands to 4× the embedding width and back, which is a lot of weights. This is also why Mixture-of-Experts targets the FFN specifically — it's where the parameters are, so it's where the savings are.
Slide 12 — The transformer block
The paper is post-norm; essentially everything modern is pre-norm, as drawn on the slide. Pre-norm trains stably at depth without the learning-rate warmup ritual post-norm requires. If you're reading the original diagram alongside the slide, that's the discrepancy you're seeing.
Slide 12 — The transformer block
No — every block has its own parameters. Same shape, different weights. Empirically they also learn different jobs: early blocks look more syntactic, later ones more semantic. (Weight-shared variants like ALBERT exist, but they're the exception.)
Slide 12 — The transformer block
Real, and it's the dominant frame in interpretability research: the skip path is a shared channel that every block reads from and writes to. Attention heads move information into it, FFNs transform it, and the final layer reads it out. Thinking of it as a bus rather than a pipeline explains a lot of observed behaviour.
Slide 13 — Residuals & LayerNorm
Batch statistics are meaningless when sequences have different lengths and are padded, and they couple examples that should be independent. LayerNorm normalises across the features of a single token, so it doesn't care what else is in the batch — essential for variable-length text.
Slide 13 — Residuals & LayerNorm
LayerNorm with the mean-subtraction dropped, so just the scaling. Slightly cheaper, works about as well, and it's what most current models use. If you see it in a config file, mentally substitute LayerNorm.
Slide 13 — Residuals & LayerNorm
One objective, unlimited data, and it turns out you can phrase almost any task as text continuation — including classification, translation and summarisation, which used to need dedicated architectures. Simplicity scaled better than specialisation.
Slide 14 — Three flavors
Not at all — for embeddings, retrieval and classification at volume, a small encoder is dramatically faster and cheaper than prompting an LLM. Plenty of production search and moderation still runs on encoders, and sentence-embedding models are a live field.
Slide 14 — Three flavors
Same formula, different sources: Q from the decoder, K and V from the encoder. That's the entire difference. It's how the original encoder-decoder reads its input, and how many multimodal models bolt an image encoder onto a language model.
Slide 14 — Three flavors
Not from this stage. Pre-training gives you a text continuer that will happily complete your question with three more questions. Instruction tuning and RLHF shape it into an assistant afterwards — a different lecture, and dramatically less data than pre-training.
Slide 15 — Next-token prediction
All at once during training — every position predicts its own next token in the same forward pass, which is exactly what the causal mask makes safe. A 1,000-token sequence yields 1,000 training signals from one pass. That's the parallelism from slide 5, finally cashed in.
Slide 15 — Next-token prediction
There's real memorisation and it's actively studied — models can be made to reproduce training text. But models are far smaller than their training data, so compression forces generalisation, and they demonstrably answer questions no single document contains.
Slide 15 — Next-token prediction
Tokenizers are trained on corpora that skew English, so other scripts fragment into more tokens per word — sometimes 2–3× the cost for the same sentence, and worse for scripts far from Latin. A real and much-discussed inequity, not a bug in your code.
Slide 16 — Tokenization
People are trying — byte-level and tokenizer-free models exist. The cost is much longer sequences, which runs straight into the n² wall from slide 6. It's an active research trade-off rather than a solved problem waiting on engineering effort.
Slide 16 — Tokenization
No — it's fit first, on a corpus sample, then frozen. Change the tokenizer and you have to retrain the model from scratch, which is why they're so sticky and why vocabulary choices made early haunt a model family for years.
Slide 16 — Tokenization
O(1) in sequential depth, not in total work. The work is still O(n²); the point is that all of it is parallelisable, so with enough hardware the wall-clock time doesn't grow with sequence length. An RNN's O(n) steps can't be parallelised at any price.
Slide 17 — Why transformers won
Yes — inference is inherently sequential, one token at a time, because each token depends on the last. That's precisely why KV-caching and speculative decoding exist. The parallelism win is a training-time win.
Slide 17 — Why transformers won
Not especially. They're compute-efficient, not data-efficient — arguably less sample-efficient than an RNN at small scale, since they have to learn locality and order from scratch. The win is that they keep improving when you add hardware, and RNNs don't.
Slide 17 — Why transformers won
It's held over many orders of magnitude. There's an irreducible-entropy floor in the maths — text has genuine unpredictability you can never model away — but nobody has hit it. The practical wall right now is data, not compute: high-quality text is finite.
Slide 18 — Scaling
Loss falls smoothly, but specific capabilities can appear to switch on at a threshold. There's genuine argument about how much of that is real and how much is an artefact of measuring with pass/fail metrics — a task scored 0 or 1 looks like a step change even when the underlying probability rose gradually.
Slide 18 — Scaling
Pre-training scaling has slowed against a data wall, and the frontier has partly shifted to inference-time compute — reasoning models that think for longer rather than models that are bigger. Worth presenting as genuinely contested rather than settled either way.
Slide 18 — Scaling
No — Flash Attention and GQA come free with current PyTorch and Hugging Face. Know the names so you can read a model card and understand why one model is cheaper to serve than another; you won't be writing the CUDA kernels.
Slide 19 — State of the art
The large model verifies every drafted token and rejects any it disagrees with, falling back to its own prediction. You only save time when the draft was right — and the output distribution is provably identical to running the large model alone.
Slide 19 — State of the art
Fair challenge. Total parameters and active parameters are different numbers, and MoE models tend to advertise the flattering one. Compute per token tracks the active count — typically a fraction of the total. Memory, however, tracks the total, which is why they're expensive to serve.
Slide 19 — State of the art
Step one. Swap the tokenizer — patch-embed an image, encode audio into frames — and steps two through five don't change at all. That's the "same architecture, different tokenizer" claim from slide 2 arriving with evidence.
Slide 20 — Putting it together
The whole post-training stack: instruction tuning, RLHF, tool use, retrieval, safety training. Real products are these five steps plus a great deal of that. The five steps are the model; the rest is the product.
Slide 20 — Putting it together
Any laptop from the last five years. CPU is fine — a few minutes to a usable loss, dropping from about 4.2 to about 1.5. It's character-level and deliberately tiny.
Slide 21 — The takeaway repo
Deliberate. A character tokenizer is about ten lines; BPE is a few hundred and would dominate a 250-line repo whose job is to make the architecture legible. Swapping BPE in is one of the exercises.
Slide 21 — The takeaway repo
Karpathy's nanoGPT — same shape, production quality, reproduces GPT-2. Then his "Let's build GPT" video if you want it built in front of you. Both are on the reading slide.
Slide 21 — The takeaway repo
| Resource | What it is |
|---|---|
| Attention Is All You Need — Vaswani et al., 2017 | The original paper. Eleven pages, surprisingly readable once you've seen this lecture. |
| The Illustrated Transformer — Jay Alammar | Best visual explanation on the internet. Free. The diagrams everyone else copies. |
| Let's build GPT, from scratch — Andrej Karpathy | Two hours of live coding that builds essentially the takeaway repo. |
| The Annotated Transformer — Harvard NLP | The paper line-by-line with runnable PyTorch beside it. |
| nanoGPT — Karpathy | Production-quality cousin of the takeaway repo. Reproduces GPT-2. |
| transformer-circuits.pub — Anthropic | What's actually happening inside the weights. A rabbit hole. |
| Slide | Topic | Questions |
|---|---|---|
| 1 | Title & frame | 2 |
| 2 | Why bother | 3 |
| 3 | Sequence modeling | 2 |
| 4 | RNNs & LSTMs | 2 |
| 5 | Two fatal flaws | 3 |
| 6 | Attention Is All You Need | 2 |
| 7 | Attention, intuitively | 3 |
| 8 | Self-attention, formally | 3 |
| 9 | The attention matrix | 3 |
| 10 | Multi-head attention | 3 |
| 11 | Position | 3 |
| 12 | The transformer block | 3 |
| 13 | Residuals & LayerNorm | 3 |
| 14 | Three flavors | 3 |
| 15 | Next-token prediction | 3 |
| 16 | Tokenization | 3 |
| 17 | Why transformers won | 3 |
| 18 | Scaling | 3 |
| 19 | State of the art | 3 |
| 20 | Putting it together | 2 |
| 21 | The takeaway repo | 3 |
| 22 | Q&A and reading | 3 |
| Total | 61 |