tokenization
Before a model reads any text, it slices the text into tokens from a fixed list. Common words stay whole as a single token. Rarer words are spelled out from smaller tokens, and any symbol at all can be built from the tiniest ones. 'A single raindrop fell from a height of 1000 meters.' is ten words, but sixteen tokens.
Cutting text into tokens from a fixed dictionary, before the model reads it; each token becomes its number in that dictionary.
The model never reads letters. Its tokenizer cuts the text into tokens from a fixed dictionary, 248,077 of them for Qwen3.5-4B, and hands the model each token's number. Common words, with the space in front, are one token each (space, “single”, space, “height”); rarer words are cut into pieces (space, “rain” + “drop”); and numbers are split into single digits. That is why lesson 2's raindrop sentence is 10 words, but 16 tokens.
The dictionary was built by byte-pair encoding (BPE): start from the 256 single bytes, find the pair that appears most often in the training text, merge it into a new token, and repeat until the dictionary is full. So frequent words end up whole, and anything else can still be spelled from pieces, down to single bytes: no text is unreadable. The lab draws a lone byte as a box holding its value in hex, F0.
Qwen3.5-4B uses byte-level byte-pair encoding. Training the tokenizer starts from the 256 possible bytes and repeatedly adds the most frequent adjacent pair in a training corpus as a new token, recording each merge in order; tokenizing applies the merges in that order. Frequent strings become single tokens, so how many tokens a text takes depends on how much text like it the tokenizer saw: space, “Paris” is one token, space, “Qwen” two, space, “antidisestablishmentarianism” six. Before any merges run, a pre-splitting rule separates words, digits, and punctuation - putting every digit on its own and detaching the space before a number. That is why space, “1000” always becomes five tokens: a standalone space and four single digits. Where the "dictionary" picture breaks: its entries are not words but whatever strings were frequent, pieces like “idis” and “arian” among them.
The vocabulary has 248,077 entries: 248,044 from byte-level BPE, which starts from the 256 bytes, and 33 special tokens (tokenizer.json). BPE came to subword segmentation with Sennrich et al. (2016, Neural Machine Translation of Rare Words with Subword Units); GPT-2 made it byte-level (Radford et al. 2019). Qwen's pre-tokenizer regex isolates each digit (\p{N}), and a space before a number stands alone, so 1000 after a space is space, “1”, “0”, “0”, “0”, the three zeros all id 15. Byte-level coverage means there is no unknown token; the cost is that a token can be part of a character (🌧️ is four tokens, the first fusing the space with F0 9F). Token counts are not a neutral measure of text: space, “дождь” (Cyrillic) (Russian, "rain") is three tokens where space, “rain” is one, so the same content costs more in some languages and scripts. In lesson 2's dictionary-tokens-qwen, "A single raindrop fell from a height of 1000 meters." is 10 words and 16 tokens.