Tokens, embeddings, and position

How text becomes numbers — and how the model knows which number came first.

Tokens — chopping text into reusable pieces

A model can't operate on raw characters directly — well, it could, but you'd waste a lot of capacity making it re-learn that "the" is a recurring chunk every single time. Instead, text gets chopped into tokens: pieces that are usually shorter than a word but longer than a character, chosen so that common pieces are reused across many words.

How merges are learned (BPE in plain terms)

Most modern tokenizers are built with a process called Byte-Pair Encoding (BPE). The algorithm is dumber than it sounds:

1. Start with the rawest possible vocabulary: every individual character.
2. Look at all the text in your training corpus. Find the most common pair of adjacent characters.
3. Add that pair as a new token. From now on, treat it as one unit.
4. Repeat — find the next most common pair (now potentially involving your new token).
5. Stop when you've added enough tokens to hit your target vocabulary size.

After thousands of iterations, you end up with a vocabulary where high-frequency patterns are single tokens and low-frequency ones decompose. No human chose them; they emerge from corpus statistics.

The result: "the" is one token (cheap), "establishment" might be one token (cheap), and "antidisestablishmentarianism" splits into something like ["anti", "dis", "establishment", "arian", "ism"] — five tokens, but built from familiar parts.

The tokenizer playground

Type anything below to see what a model would actually receive.

Why tokenization is more interesting than it sounds

Tokenization is one of those topics that feels mechanical until it bites you. A few cases worth internalizing:

Numbers are often broken

A typical tokenizer might encode 1234 as a single token (because that exact sequence appeared often enough during BPE training) but 1235 as three: ["12", "3", "5"]. To the model, those two numbers don't look at all alike. This is part of why arithmetic is hard for LLMs: the input representation actively obscures numeric structure.

Multilingual cost is asymmetric

English benefits massively from BPE — it's what the vocabulary was tuned on. Other languages, especially non-Latin scripts, often pay a "tokenization tax" of 2–5× more tokens for the same semantic content. "Hello, how are you?" might be 5 tokens in English; the equivalent Hindi sentence in Devanagari script could be 25. Per-token pricing turns directly into per-language pricing.

Code has its own dialect

Tokenizers trained mostly on prose handle code awkwardly: function might be a single token (good), but HashMap<String, Integer> could be a dozen tokens (bad). Models marketed for code (Code Llama, GPT-4 family) usually have vocabularies trained on more code, which makes them not just smarter at code but cheaper to run on it.

Whitespace and capitalization matter

Most tokenizers treat " the" (with leading space) as a different token than "the" (no space), because they often appear in different contexts. "The", "the", and "THE" are usually three separate tokens too. This is invisible to humans but very visible to the model.

Embeddings — turning IDs into geometry

A token ID like 1820 isn't useful to the math directly. The model needs each token represented as a vector — a list of (typically) 1024 to 8192 numbers. That vector is the token's embedding, and it's looked up from a giant table called the embedding matrix.

The famous example: in the embedding space of GPT-2 and similar models, the vector arithmetic king − man + woman lands close to queen. Meaning has become geometry. This isn't programmed in — it's an emergent property of training the model to predict the next token across enormous amounts of text.

Position — how the model knows what came first

Here's the gap that most explanations skip too long: by themselves, embeddings carry no information about word order. "Dog bites man" and "man bites dog" would produce identical sets of embeddings — same words, same vectors. The model would have no way to tell which word came first.

So position information has to be injected separately. Several schemes exist; modern decoder-only LLMs almost universally use one called RoPE (Rotary Position Embedding).

RoPE in plain terms

RoPE works by rotating each token's query and key vectors by an angle that depends on the token's position. Same word at position 1 versus position 50 ends up rotated differently. When attention later compares two tokens (by computing the dot product of their Q and K), the rotation difference encodes the relative distance between them.

This trick has nice properties:

• Relative position is built in. Two tokens 5 apart always have the same angle difference, regardless of where in the sequence they are.
• It's parameter-free. No extra weights to learn — just a deterministic rotation formula.
• It can be "stretched" later. When you want a model trained on 4k context to handle 32k context, you can rescale the rotation frequencies (RoPE scaling — see levers.html §3). It's not perfect, but it's better than retraining.