Glossary

Each term is explained at four levels, from Simple to Expert; an entry's slider moves between them.

autoregression

Writing one token at a time, each time adding the token to the text and running the model again on the longer text.

embedding

Each token's ID number picks one row from a big lookup table called the embedding table.

logits

Logits are the raw 'scores' from lesson 1, before softmax turns them into odds.

prefilling and decoding

The two speeds of running a model: prefilling reads the whole prompt in one pass; decoding then writes one token per pass.

sampling

The model only gives odds; a separate step has to pick the winning token.

softmax

Softmax is a mathematical operation that turns a list of scores into probabilities that add up to 100%.

temperature

A setting that sharpens the odds (below 1) or flattens them (above 1) before a token is sampled; at 1 they are the model's own.

tied weights

A model needs a table to turn tokens into rows of numbers on the way in (the embedding table) and a table to score its output against every token on the way out.

token

A token is a chunk of text from the model's fixed vocabulary, which is 248,077 entries in Qwen3.5-4B.

tokenization

Cutting text into tokens from a fixed dictionary, before the model reads it; each token becomes its number in that dictionary.