TutorialDay 004Transformers7 steps~4 minOnly needs eˣ
Attention, by hand
Work out where a transformer looks, using three tokens and a calculator.
Read the slides as text 7 slides
Slide 1
Attention, by hand
Work out where a transformer looks, using three tokens and a calculator.
7 steps
~4 min
Only needs eˣ
Slide 2 · THE BIG PICTURE
The whole formula is three moves
Attention(Q,K,V) = softmax( QKᵀ/√dₖ ) V
Vaswani et al., "Attention Is All You Need" (2017), Eq. 1
1. Score
Dot product of the query with every key, divided by √dₖ. One number per token: how well it matches.
q · k ÷ √dₖ
2. Normalise
Softmax: raise e to each score and divide by the total. Weights are positive and add up to 1.
[3, 1, 0] → [.84, .11, .04]
3. Mix
Multiply each token's value vector by its weight and add them up. That sum is the output.
Σ wᵢ · vᵢ
Q query: what the current token is looking for
K key: what each token offers to be matched on
V value: what each token passes on if picked
Slide 3 · STEP 1 · SCORE
Score every token against the query
Toy setup: each statement squeezed into one token · dₖ = 4 · one head · no positions or learned projections.
What is the secret code of the river?
QUERY VECTOR
q = [2, 0, 1, 1]
| TOKEN | KEY K | Q · K | ÷ √4 | SCORE |
|---|---|---|---|---|
| river's code is 4821 | [2, 0, 1, 1] | 6 | ÷ 2 | 3 |
| lamp's code is 7390 | [0, 2, 1, 1] | 2 | ÷ 2 | 1 |
| the weather was mild | [0, 1, 0, 0] | 0 | ÷ 2 | 0 |
WORKED: THE RIVER FACT
2×2 + 0×0 + 1×1 + 1×1 = 6
6 ÷ √4 = 6 ÷ 2 = 3WHY DIVIDE BY √dₖ?
Dot products grow with vector size. Large scores push softmax into a region where gradients get tiny (Vaswani et al., §3.2.1).
Slide 4 · STEP 2 · SOFTMAX
Turn scores into a 100% budget
| TOKEN | SCORE | E^SCORE (ITS PULL) | SHARE |
|---|---|---|---|
| river's code is 4821 | 3 | 20.09 | 84.4% |
| lamp's code is 7390 | 1 | 2.72 | 11.4% |
| the weather was mild | 0 | 1.00 | 4.2% |
total = 20.09 + 2.72 + 1.00 = 23.80 · share = e^score ÷ 23.80
WHERE THE ATTENTION GOES
84.4% river · 4821 11.4
Every token gets a slice; the slices always add up to 100% 4.2% weather ↗
84.4%
ON THE EVIDENCE
of the attention budget goes to the river fact
RULE OF THUMB
+1 score = ×2.72
pull = e^score, and e ≈ 2.718
Slide 5 · STEP 3 · MIX
Blend the values using those weights
| TOKEN | WEIGHT | VALUE V |
|---|---|---|
| river's code is 4821 | 0.844 | [1, 0, 0] |
| lamp's code is 7390 | 0.114 | [0, 1, 0] |
| the weather was mild | 0.042 | [0, 0, 0] |
Value slots = answers [4821, 7390, 1937]. The weather line carries no answer.
THE OUTPUT
0.844·[1,0,0]
+ 0.114·[0,1,0]
+ 0.042·[0,0,0]
= [0.84, 0.11, 0.00]OUTPUT, BY ANSWER
0.84 4821 0.11 7390 0.00 1937
✓ The model leans hard towards 4821.
Slide 6 · TWIST · ADD A RIVAL
One lookalike cuts the evidence to 46%
NEW TOKEN river's code is 1937 k = [2, 0, 1, 1] → score 3, same as the real fact
BEFORE
84.4% · 4821 11.4
AFTER
45.8% · 4821 45.8% · 1937
Output is now [0.46, 0.06, 0.46]: 4821 and 1937 are tied. A coin flip.
QUICK CHECK
Filler tokens score 0. How many would it take to pull as much attention as one rival scoring 3?
- A. 1
- B. 3
- C. about 20
- D. about 400
Slide 7
Answer: C, about 20
A token's pull is e^score, so one rival = e³ ÷ e⁰ = 20.1 filler tokens.
SAME PULL
1 rival =
Bonus: if the model scores filler at −3, you'd need e⁶ ≈ 403 of them.
TAKEAWAY
Attention is a fixed budget split by e^score. Whatever looks most like the answer takes the biggest share, right or wrong.
CHECK IT IN CODE
import torch
scores = torch.tensor([3., 1., 0.])
torch.softmax(scores, dim=0)
# tensor([0.8438, 0.1142, 0.0420])WHERE THIS SHOWS UP · DAY 003
Liu (2026): attention left on the evidence explained up to 58% of the variance in retrieval loss. Log length explained at most 8%.
REFERENCES
arxiv.org/abs/1706.03762
pytorch.org/docs · scaled_dot_product_attention
arxiv.org/abs/2611.04169