The Shared Dictionary

What the picture shows

Lesson 1’s picture again: the question ‘What is the capital of France?’ goes into the closed box, Qwen3.5-4B, and the reply ‘The capital of France is **Paris**.’ comes out token by token. Then the lid comes off to show what the box is made of: numbers, drawn as 32 columns of points, one per layer, labelled 4,205,751,296 numbers.

The next-token loop of mlx-community/Qwen3.5-4B-4bit on the question “What is the capital of France?” wrapped in the chat template, 19 tokens: “<|im_start|>” (special token), “user”, line break, “What”, space, “is”, space, “the”, space, “capital”, space, “of”, space, “France”, “?” (punctuation or formatting), “<|im_end|>” (special token), line break, “<|im_start|>” (special token), “assistant”, line break, “<think>” (special token), 2 line breaks, “</think>”, 2 line breaks. At each step, the three most likely next tokens: step 1: “The” 100%, “**” (punctuation or formatting) <0.1%, “Paris” <0.1%; chose “The”; step 2: space, “capital” 100%, space, “national” <0.1%, space, “**” (punctuation or formatting) <0.1%; chose space, “capital”; step 3: space, “of” 100%, space, “city” <0.1%, space, “and” <0.1%; chose space, “of”; step 4: space, “France” 100%, space, “**” (punctuation or formatting) <0.1%, space, “the” <0.1%; chose space, “France”; step 5: space, “is” 100%, space, “(” (punctuation or formatting) <0.1%, “,” (punctuation or formatting) <0.1%; chose space, “is”; step 6: space, “**” (punctuation or formatting) 100%, space, “Paris” 0.2%, space, “the” <0.1%; chose space, “**” (punctuation or formatting); step 7: “Paris” 100%, space, “Paris” <0.1%, “巴黎” (Chinese characters) <0.1%; chose “Paris”; step 8: “**.” (punctuation or formatting) 95%, “**” (punctuation or formatting) 4.7%, “**,” (punctuation or formatting) <0.1%; chose “**.” (punctuation or formatting).

Qwen3.5-4B: 32 layers, 2560 numbers wide, 248,077 tokens in its vocabulary, 4,205,751,296 parameters in 2,367,118,848 bytes, quantised to 4 bits in groups of 64.