01 · Row × column列 × 欄
Reproduce the lecture example重做講義範例：把矩陣的一列乘上向量
W = [(1, 2); (1, −1); (−1, 1)], x = (2, 1)ᵀ.
Compute all three scores. Write each as a sum of two products.算出三個分數。每個分數都寫成兩個乘積的和。
Tokens → vectors詞元 → 向量Attention → weighted sum注意力 → 加權和Wx → scores → probabilities分數 → 機率
Suggested beginner route高中生建議路線
- Pick a token and move its two coordinate sliders to see how the vector changes.先選一個詞元（token），移動兩個座標滑桿，看看向量如何改變。
- Press “Start with Wx” and work with the simple x=(2,1). You can skip attention for now.按「從矩陣範例開始」，先算簡單的 x=(2,1)，不用先看注意力。
- Do the two exercises on multiplying row by row and on one matrix entry. To see where x comes from, return to attention (optional).做「逐列乘法」與「一個矩陣元素」兩題。想知道 x 如何產生，再回到注意力（選讀）。
Predict → change one value → inspect the calculation → read the key idea.先預測 → 改一個值 → 檢查算式 → 看懂重點概念。
Each token starts as a vector.每個詞元（token）一開始都是一個向量。
A token embedding is a learned list of numbers. Tokens need not be whole words: a word split into three tokens receives three initial vectors.詞元的嵌入向量（embedding）是一串經由訓練學到的數字。詞元不一定是完整的單字：一個字若被切成三個詞元，就會得到三個初始向量。
one word一個字 → (t₁, t₂, t₃) → (e(t₁), e(t₂), e(t₃)), e(t) ∈ ℝ² here.（本例）。
Choose a token, then change its coordinates選一個詞元，再改它的座標
Changing the embedding of the changes both occurrences. They share an initial vector, but attention can give their positions different processed vectors.改變 the 的嵌入向量，兩個 the 會一起改變。它們共用同一個初始向量，但注意力可以讓這兩個位置得到不同的處理後向量。
The coordinates here are invented. Individual coordinates in a trained LLM do not usually have a simple name such as “catness.”這裡的座標是虛構的。訓練好的大型語言模型（LLM）中，單一座標通常沒有「像貓的程度」這類簡單的名稱。
How does this relate to training?這和訓練有什麼關係？
Training adjusts an embedding table and other model weights using many prediction examples. The controls here edit those numbers by hand; this simulator does not train a model. During ordinary generation the weights stay fixed while the input changes.
訓練會用大量的預測例子，調整嵌入向量表和模型的其他權重（weight）。這裡的控制項是用手修改這些數字；這個模擬器不會訓練模型。一般產生文字時，權重固定不變，改變的只有輸入。
See the vector in ℝ²在 ℝ² 中看這個向量
Use either slider to move the vector. The two coordinates are its horizontal and vertical components.用任一個滑桿移動向量。兩個座標就是它的水平分量與鉛直分量。
| Position / token位置／詞元 | dim 1維度 1 | dim 2維度 2 |
|---|
Toy tokenization: lowercase, split at spaces; each displayed word is one token.玩具斷詞方式：全部小寫、以空格切開；每個顯示的單字就是一個詞元。
Start with the weighted sum選讀：先看權重與加權和The attention weights in each row add up to 1; × marks a future position that cannot be seen. Click a table cell to inspect one dot product, then see in the card “The last position supplies our context vector” how the vectors are mixed into x. The matrix editor can be expanded below.每一列的注意力權重（attention weight）加總為 1；× 表示不能看未來的位置。先點表格看一個內積（dot product），再看「最後一個位置提供上下文向量」卡片如何把向量混合成 x。下方可展開矩陣編輯器。
Attention mixes vectors from the available context.注意力（attention）會混合上下文中可用的向量。
Three matrices turn each embedding into a query (what this position compares), a key (what other positions are compared by), and a value (the information to mix). These are helpful roles, not literal human questions.三個矩陣把每個嵌入向量轉成查詢（query）（這個位置拿來比較的向量）、鍵（key）（其他位置被比較時用的向量）與值（value）（要混合的資訊）。這些只是方便理解的角色，並不是真的像人在提問。
qᵢ = AQe(tᵢ), kⱼ = AKe(tⱼ), vⱼ = AVe(tⱼ). Each A is 2 × 2.每個 A 都是 2 × 2 矩陣。
Optional: edit the query, key and value matrices選讀：修改查詢、鍵、值的矩陣
Query vector q: the vector that does the comparing. Key vector k: the vector being compared. Value vector v: the information mixed by the weights. You can keep the default values for now.查詢向量（query）q：拿來比較的向量。鍵向量（key）k：被比較的向量。值向量（value）v：依權重混合的資訊。先保留預設值即可。
Rows select output coordinates; columns select input coordinates. The initial matrices are identities. AV is the value projection; it is different from the output scoring matrix W in step 3.列（row）對應輸出的座標；欄（column）對應輸入的座標。初始矩陣都是單位矩陣。AV 是值的投影矩陣，和步驟 3 用來計算分數的輸出矩陣 W 不同。
See one matrix multiplication in full完整看一次矩陣乘法
Dot products → attention weights內積（dot product）→ 注意力權重
Inspect any query / key pair檢查任一組查詢／鍵
Rows are query positions i; columns are key positions j. Click a cell to inspect its calculation. Future positions are masked.列是查詢位置 i；欄是鍵位置 j。點一格就能看它的計算過程。未來的位置會被遮住（因果遮罩，causal mask）。
For當 j ≤ i:時， αᵢⱼ = exp(zᵢⱼ) / ∑ℓ≤i exp(zᵢℓ), where，其中 zᵢⱼ = qᵢᵀkⱼ / √2. For。當 j > i:時， αᵢⱼ = 0.
The exponential makes weights positive; division makes every row sum to 1. The √2 factor uses the query/key dimension, which is 2 here.softmax 中的 exp(z)=ez 先把每個數變成正數，再除以這些正數的總和；因此權重都是正的，而且每一列加總為 1。不需要先學微積分。除以 √2 是因為查詢／鍵向量的維度在這裡是 2。
The last position supplies our context vector最後一個位置提供上下文向量（context vector）
Context vector上下文向量computed, not guessed算出來的，不是猜的
Here we set x = cₙ, the last attention output. A full Transformer processes its vectors through additional operations and layers.這裡取 x = cₙ，也就是最後一個位置的注意力輸出。完整的 Transformer 還會讓向量經過更多運算與更多層。
Pack the vectors into matrices把向量排成矩陣
E = [e(t₁) … e(tₙ)] ∈ ℝ2×n
Q = AQE, K = AKE, V = AVE
Z = QᵀK / √2 ∈ ℝn×n
A = row-softmax(causal-mask(Z))A = 逐列 softmax(因果遮罩(Z))
C = VAᵀ ∈ ℝ2×n, x = C[:, n].
Q = AQE, K = AKE, V = AVE
Z = QᵀK / √2 ∈ ℝn×n
A = row-softmax(causal-mask(Z))A = 逐列 softmax(因果遮罩(Z))
C = VAᵀ ∈ ℝ2×n, x = C[:, n].
We store vectors as columns, matching the lecture. A contains attention weights; AQ, AK, AV are separate projection matrices.和講義一致，我們把向量寫成欄向量。A 裝的是注意力權重；AQ、AK、AV 則是另外的投影矩陣。
One row, one candidate一列對一個候選詞元Under “Where does x come from?”, choose the lecture example x = (2, 1) first. The row for “mat” is (1,2), so its score is 1×2+2×1=4. Use the buttons to check each term, then change one weight or coordinate and predict how the score will change.先在「x 從哪裡來？」選「講義範例：x = (2, 1)」。mat 這列是 (1,2)，所以分數為 1×2+2×1=4。按按鈕逐項檢查，再改一個權重或座標，先猜分數會如何改變。
One row, one token score.一列對應一個詞元的分數。
Multiply each row of W by the context vector. Each entry tells us how strongly one coordinate contributes to that candidate’s score.把 W 的每一列和上下文向量做內積（dot product）。每個元素表示某一個座標對該候選詞元分數的貢獻有多大。mat 是「地墊」，rug 是「地毯」，moon 是「月亮」。
s = Wx, W ∈ ℝ3×2, x ∈ ℝ², s ∈ ℝ³. sᵢ = Wᵢ₁x₁ + Wᵢ₂x₂.
Recall Chapter 3回想第 3 章W is a 3×2 matrix: a function that sends a vector in ℝ² to three scores in ℝ³. Read by columns, Wx = x₁·(column 1) + x₂·(column 2), so increasing x₂ by 1 adds column 2 to the scores. Review matrix transformations.W 是 3×2 矩陣：把 ℝ² 的向量送到 ℝ³ 的三個分數的函數。用欄來讀，Wx = x₁·（第 1 欄）+ x₂·（第 2 欄），所以 x₂ 增加 1，分數就加上第 2 欄。複習矩陣即線性變換。
Your calculator (editable) · input vector你的計算器（可修改）· 輸入向量
Editable output matrix W輸出矩陣 W（可修改）
| Candidate / row候選詞元／列 | x₁ columnx₁ 欄 | x₂ columnx₂ 欄 |
|---|
Select a row to unpack its dot product. Edit a weight to change the rule; change x to use the same rule on different context.選一列，拆開看它的內積。改一個權重就是改變規則；改 x 則是把同一個規則用在不同的上下文。
In a full vocabulary output layer, the number of rows equals the vocabulary size (tokens, not necessarily words). Here we score only three candidates; the input token list is larger.在涵蓋完整詞彙表的輸出層中，列數等於詞彙表大小（以詞元計，不一定是單字）。這裡只替三個候選詞元計分；輸入的詞元清單比較長。
Multiply, then add先乘，再相加
Scores and probabilities分數與機率
| Token詞元 | Score分數 | Probability*機率* |
|---|
pᵢ = exp(sᵢ) / (exp(smat) + exp(srug) + exp(smoon)).
*Normalized over these three candidates only. Scores can be negative; probabilities are nonnegative and sum to 1. The highest score is a possible choice, not the only way real systems select a token.*只在這三個候選詞元之間用 softmax 正規化。分數可以是負的；機率非負，而且加總為 1。選分數最高的詞元是一種可能的做法，但真實系統不一定只用這種方式挑選詞元。
Challenge · lecture W (fixed)挑戰 · 講義的 W（固定）
A score is a dot product: think in directions分數就是內積：用方向來想
This uses the lecture W: each candidate token is one row of W, drawn as an arrow. Drag the red x and watch the three scores. For a fixed row and fixed length of x, a smaller angle gives a larger dot product. Comparing different rows also requires their lengths: w·x=‖w‖‖x‖cosθ. Perpendicular gives 0; an obtuse angle gives a negative score.用講義的 W：三個候選詞元各對應 W 的一列，畫成三支箭頭。拖動紅色的 x，看三個分數怎麼變。固定某一列及 x 的長度時，夾角越小，內積越大。比較不同列的分數還要考慮長度：w·x=‖w‖‖x‖cosθ。垂直時是 0；夾角為鈍角時是負的。
x =
Tasks任務
Why linear algebra matters: vectors represent tokens, matrix products create queries, keys and values, and weighted sums combine context. Finally, Wx turns a context vector into token scores. Softmax is nonlinear, so the whole model is not just one linear map.為什麼線性代數重要：向量代表詞元，矩陣乘法產生查詢、鍵與值，加權和把上下文組合起來。最後，Wx 把上下文向量變成詞元的分數。softmax 是非線性的，所以整個模型並不只是一個線性映射。
Core first, then extend先做前兩題；後兩題是注意力延伸The first two exercises need only multiplication and addition. Leave the third (averaging) and the fourth (a limitation of the model) until after the optional attention section. A wrong answer shows a calculation hint.前兩題只需乘法和加法。第三題的平均、第四題的模型限制，可在完成注意力選讀後再做。答案錯誤時會有計算提示。
Predict first. Change one thing. Verify by hand.先預測，一次只改一個值，再動手驗算。
Allow 20–25 minutes. You need addition, multiplication and vectors with two coordinates. Make a prediction before checking it; the evidence is the calculation, not agreement with the screen.預留 20–25 分鐘。你需要會加法、乘法，以及有兩個座標的向量。檢查之前，先做預測；證據是算式本身，而不是和螢幕上的結果一致。
02 · Read a matrix entry讀懂矩陣的一個元素
What does W₁₂ = 2 do?W₁₂ = 2 有什麼作用？第二個座標增加 1，分數會增加多少？
Keep the same W and x₁ = 2. Increase x₂ from 1 to 2. Predict the change in the mat score.W 不變，x₁ = 2。把 x₂ 從 1 增加到 2，預測 mat 分數的改變量。
Generalize after your prediction預測之後，再推廣到一般情形
sᵢ(x + Δx) − sᵢ(x) = Wᵢ₁Δx₁ + Wᵢ₂Δx₂.
With Δx₁ = 0 and Δx₂ = 1, the score changes by Wᵢ₂. Verify the rug and moon changes too.當 Δx₁ = 0、Δx₂ = 1 時，分數改變 Wᵢ₂。也驗證 rug 和 moon 分數的改變量。
03 · A weighted sum of vectors向量的加權和
Make attention uniform讓注意力平均分配：每個權重相同時，就是取平均
Use the original cat prefix and embeddings. Set AQ = 0 and AK = AV = I. All five last-position dot products become 0.使用原本的 cat 前文與嵌入向量。設 AQ = 0、AK = AV = I，最後一個位置的五個內積都會變成 0。
Predict αₙⱼ, then compute the average of (0,1), (2,1), (1,0.4), (0,0.2), (0,1).先預測 αₙⱼ，再計算 (0,1)、(2,1)、(1,0.4)、(0,0.2)、(0,1) 的平均。
04 · Critique the model檢討這個模型
Can this toy model really read?這個玩具模型真的會閱讀嗎？找出它忽略語序的限制
Compare “The cat sat on the” with “on sat cat the the”. Predict whether their final context vectors differ. Then change the prefix selector and compare x.比較「The cat sat on the」和「on sat cat the the」。先預測它們最後的上下文向量是否不同，再切換前文選單，比較 x。
Reveal the explanation after testing測試之後再看解釋
They give the same final x (up to rounding), because the earlier tokens are the same collection, the last token is unchanged, and this one-head toy omits positional information. Reordering only permutes the terms in the final weighted sum.
兩者最後得到相同的 x（只差在四捨五入），因為前面的詞元是同一組、最後一個詞元沒變，而且這個只有一個注意力頭的玩具模型省略了位置資訊。重新排序只是把最後加權和中的各項換個順序。
A real Transformer needs a way to represent position. This counterexample shows why numerical output alone is not evidence of language understanding.
真正的 Transformer 需要表示位置的方法。這個反例說明：只看數值輸出，不能當作模型理解語言的證據。
Reflection: Why does the linearity of Wx not make the entire attention calculation linear?反思：Wx 是線性的，為什麼不代表整個注意力計算也是線性的？
Check the linearity distinction核對線性的差別
For fixed W, W(ax + by) = aWx + bWy. But attention weights depend on q and k through dot products and softmax. The weights are not fixed when embeddings change, so the complete attention map is generally nonlinear.當 W 固定時，W(ax + by) = aWx + bWy。但注意力權重透過內積與 softmax 由 q 和 k 決定。嵌入向量改變時，權重也會跟著改變，所以完整的注意力映射一般是非線性的。
Your evidence: one hand-calculated dot product, one predicted score change, one checked weighted average, and one limitation of the toy model. No generative AI account or external service is needed.你的學習證據：一個親手算出的內積、一個預測的分數改變量、一個核對過的加權平均，以及這個玩具模型的一個限制。不需要任何生成式 AI 帳號或外部服務。
Instructor guide & model boundaries教師指引與模型限制
- Start in step 3 with the lecture vector (2,1). Ask students to calculate [4,1,−1] before revealing the answer.從步驟 3 的講義向量 (2,1) 開始。揭曉答案前，先請學生算出 [4,1,−1]。
- Use step 1 to distinguish tokens, vector dimension and sequence length. A repeated token shares its initial embedding.用步驟 1 區分詞元、向量維度與序列長度。重複出現的詞元共用同一個初始嵌入向量。
- Open step 2 and follow q → qᵀk → α → x. Check that each attention row sums to 1 and masked cells have weight 0.打開步驟 2，依序追蹤 q → qᵀk → α → x。確認每一列注意力權重加總為 1，且被遮住的格子權重為 0。
- Switch the prefix while keeping all matrices fixed, then return to step 3. Check which quantities changed.固定所有矩陣、只切換前文，再回到步驟 3。看看哪些量改變了。
- Finish with a prediction exercise and the model-limitation exercise. The fixed numbers in the exercises can also be checked by hand.最後做一題預測練習，再看模型限制那一題。練習中的固定數字也可以用手算核對。
This deterministic educational model uses invented, untrained 2D embeddings, one masked self-attention head, and a restricted three-candidate output matrix. It omits positional encodings, residual connections, normalization, feed-forward layers, multiple heads/layers, biases and training. It does not reproduce GPT-3 responses. Changing a number is an intervention, not a training step.
這是一個確定性的教學模型：使用虛構、未經訓練的二維嵌入向量、一個加上遮罩的自注意力頭（self-attention head），以及只含三個候選詞元的輸出矩陣。它省略了位置編碼、殘差連接、正規化、前饋層、多頭／多層、偏差項與訓練，也不會重現 GPT-3 的回應。改一個數字是一次人為介入，不是一次訓練步驟。