Tokenisation and Embeddings

Before a model can attend to anything, text must become integers and integers must become vectors. You will build a byte-pair encoder from scratch, train it on English and Bahasa Melayu, see why “makanan” and “minuman” split so differently, and turn token ids into embedding rows with NumPy.

Ask a language model how many letters r there are in “strawberry” and, until quite recently, many got it wrong. The reason was not a lack of intelligence in the usual sense. The model never saw the letters. It saw a short list of integers, each standing for a chunk of text called a token, chosen by a separate algorithm before training began, and the letters inside each chunk were hidden from it. That algorithm, the tokeniser, is the least glamorous part of a language model and one of the most consequential. It decides what the model can easily see, how much a sentence in Bahasa Melayu costs compared with the same sentence in English, and why arithmetic on long numbers was once so shaky. This page builds one.

Why not characters, or words?

A Transformer needs its input as a sequence of integers drawn from a fixed vocabulary, each of which indexes a row of an embedding table. There are two obvious ways to choose that vocabulary, and both fail.

Characters. The vocabulary is tiny (a few hundred symbols for most scripts) and nothing is ever unknown. But sequences become long: a 1,000-word document is roughly 6,000 characters. Attention costs grow with the square of the length, and the model must spend its early layers rebuilding words from letters before it can think about meaning. Character-level models work — you will train one on the pretraining page — but they are inefficient at scale.

Words. Sequences are short and each unit carries meaning. But the vocabulary explodes. English alone has hundreds of thousands of word forms; add names, typos, code, URLs and other languages and no fixed list suffices. Any word missing from the list becomes a single “unknown” token and its meaning is lost. Malay makes the problem vivid: from the root makan (eat) come makanan, memakan, dimakan, pemakanan, makan-makan and more. A word-level vocabulary must store each separately and learns nothing about one from the others.

Subwords are the compromise every major language model uses As of 2026. Common words get a single token; rarer words are built from a few common pieces; and the alphabet (or the set of bytes) is always available as a fallback, so nothing is unknown. Typical English text comes out at roughly three-quarters of a word per token, though that ratio depends on the tokeniser and the text.

Byte-pair encoding, by hand

The most widely used subword method is byte-pair encoding (BPE). It began as a data-compression trick published by Philip Gage in 1994; Rico Sennrich, Barry Haddow and Alexandra Birch adapted it for neural machine translation in 2016, and GPT-2 made a byte-level version standard for language models. The training algorithm is greedy and almost embarrassingly simple:

  1. Start from the smallest unitsSplit every word of the training text into characters (or bytes). The initial vocabulary is just that alphabet.
  2. Count adjacent pairsAcross the whole corpus, count how often each pair of neighbouring symbols occurs, such as (a, n) or (m, a).
  3. Merge the most frequent pairReplace every occurrence of the winning pair with a new single symbol (a + n → an) and add it to the vocabulary. Record the merge.
  4. RepeatRecount and merge again, thousands or hundreds of thousands of times, until the vocabulary reaches the size you chose. Frequent words end up as single symbols.
  5. Encode new text with the recorded mergesTo tokenise new text, split it into units and apply the learned merges in the same order. The merge list is the tokeniser.

Two properties follow. First, the result depends entirely on the training corpus: what is frequent there becomes cheap, what is rare stays fragmented. Second, the vocabulary size is a dial you set, not something the algorithm discovers.

BPE in thirty lines

Here is the algorithm on a small mixed text, mostly Bahasa Melayu with some English. To keep merges inside words (as most production tokenisers do, using a pre-splitting step), the text is split on spaces and each word is counted. An underscore marks the start of a word, playing the role of the leading space that real tokenisers keep.

from collections import Counter

text = ("saya makan nasi lemak dan minum teh tarik . kami makan makanan "
        "sedap dan minum minuman sejuk . mereka makan nasi lemak lagi . "
        "we eat nasi lemak and we drink teh tarik . they eat and they drink .")

# each word starts as a tuple of characters, with "_" marking a word start
words = Counter(tuple("_" + w) for w in text.split())

def pair_counts(words):
    pairs = Counter()
    for w, n in words.items():
        for a, b in zip(w, w[1:]):
            pairs[a, b] += n
    return pairs

def merge(w, pair):
    out, i = [], 0
    while i < len(w):
        if i < len(w) - 1 and (w[i], w[i + 1]) == pair:
            out.append(w[i] + w[i + 1]); i += 2
        else:
            out.append(w[i]); i += 1
    return tuple(out)

merges = []
for step in range(15):
    pairs = pair_counts(words)
    best, count = pairs.most_common(1)[0]
    merges.append(best)
    words = Counter({merge(w, best): n for w, n in words.items()})
    print(f"merge {step + 1:>2}: {best[0]!r:>8} + {best[1]!r:<6} -> {best[0] + best[1]!r:<10} ({count}x)")

def encode(word):
    w = tuple("_" + word)
    for pair in merges:                      # apply merges in the order learned
        w = merge(w, pair)
    return w

for w in ["makanan", "minuman", "lemak", "makan-makan", "drinking"]:
    print(f"{w:<12} -> {encode(w)}")
merge  1:      'a' + 'n'    -> 'an'       (10x)
merge  2:      '_' + 'm'    -> '_m'       (8x)
merge  3:      'a' + 'k'    -> 'ak'       (7x)
merge  4:      '_' + 't'    -> '_t'       (6x)
merge  5:      'i' + 'n'    -> 'in'       (5x)
merge  6:      '_' + '.'    -> '_.'       (5x)
merge  7:     '_m' + 'ak'   -> '_mak'     (4x)
merge  8:   '_mak' + 'an'   -> '_makan'   (4x)
merge  9:      '_' + 'l'    -> '_l'       (4x)
merge 10:      '_' + 'd'    -> '_d'       (4x)
merge 11:      '_' + 's'    -> '_s'       (3x)
merge 12:      '_' + 'n'    -> '_n'       (3x)
merge 13:     '_n' + 'a'    -> '_na'      (3x)
merge 14:    '_na' + 's'    -> '_nas'     (3x)
merge 15:   '_nas' + 'i'    -> '_nasi'    (3x)
makanan      -> ('_makan', 'an')
minuman      -> ('_m', 'in', 'u', 'm', 'an')
lemak        -> ('_l', 'e', 'm', 'ak')
makan-makan  -> ('_makan', '-', 'm', 'ak', 'an')
drinking     -> ('_d', 'r', 'in', 'k', 'in', 'g')

Read the merges as the corpus’s statistics made visible. The first, a + n → an, wins because “an” appears ten times (in makan, makanan, minuman, dan, and). Step by step the algorithm assembles _mak, then _makan as a single token after eight merges, because makan occurs four times as a word or word-start. _nasi takes four merges in a row (_n, _na, _nas, _nasi). When counts tie, Python’s Counter.most_common returns the pair it saw first, so the order is deterministic.

Now the encodings. makanan splits cleanly as _makan + an — root plus suffix, which is linguistically sensible, though nobody told the algorithm about Malay morphology; it emerged from frequency. minuman was seen only once, so it is still five pieces. drinking never appeared and falls back to six small pieces. This is the subword bargain: frequent things are cheap, rare things are expensive but never impossible. Real tokenisers run the same loop with tens of thousands of merges over billions of words; Karpathy’s minbpe is an excellent, readable reference implementation, and production libraries add speed (merging by priority in a single pass), special tokens and careful pre-splitting rules.

The encoder above is the simplest correct version: it replays every merge in order. That is slow for long texts, which is why real implementations instead repeatedly find the adjacent pair with the earliest merge rank and apply it — the same result, far fewer passes.

Bytes, not characters

Character-level BPE still has an “unknown” problem: a character never seen in training — an emoji, a rare Chinese character — has no symbol. GPT-2’s answer (Radford and colleagues, 2019) was to run BPE over the UTF-8 bytes of the text instead of its characters. There are only 256 possible bytes, so every string, in any script, can always be encoded. GPT-2’s vocabulary was exactly that: 256 byte tokens, 50,000 learned merges, and one special end-of-text token, for 50,257 in total.

samples = ["hello", "terima kasih", "谢谢", "நன்றி", "café", "👍"]
for s in samples:
    b = s.encode("utf-8")
    print(f"{s!r:<16} {len(s):>2} chars -> {len(b):>2} bytes  {list(b)[:6]}")
'hello'           5 chars ->  5 bytes  [104, 101, 108, 108, 111]
'terima kasih'   12 chars -> 12 bytes  [116, 101, 114, 105, 109, 97]
'谢谢'              2 chars ->  6 bytes  [232, 176, 162, 232, 176, 162]
'நன்றி'           5 chars -> 15 bytes  [224, 174, 168, 224, 174, 169]
'café'            4 chars ->  5 bytes  [99, 97, 102, 195, 169]
'👍'               1 chars ->  4 bytes  [240, 159, 145, 141]

ASCII letters take one byte each, so English and romanised Malay start cheap. “Café” needs an extra byte for the accented é. The two Chinese characters take three bytes each; the five Tamil code points (“நன்றி”, thank you) take fifteen; a single emoji takes four. In byte-level BPE, those bytes are the raw material. If the training corpus contained plenty of Tamil, merges will have glued common Tamil byte sequences into whole tokens; if it contained little, a Tamil word may cost a token per byte or close to it. (If the printout fails on a Windows console with an encoding error, set the environment variable PYTHONIOENCODING=utf-8.)

Byte-level BPE is not the only approach. Google’s SentencePiece library (Kudo and Richardson, 2018) treats text as a raw stream including spaces and supports both BPE and a probabilistic “unigram” method (Kudo, 2018); BERT used WordPiece, a close cousin of BPE that chooses merges by a likelihood score rather than raw counts. The family resemblance is strong: learn a subword vocabulary from data, fall back to small units when needed.

How big should the vocabulary be?

Vocabulary size is a trade-off with costs on both sides.

d_model = 4096
for vocab in [32_000, 100_000, 256_000]:
    table = vocab * d_model
    print(f"vocab {vocab:>7,}: embedding table {table / 1e6:>7,.0f}M parameters "
          f"(x2 if the output layer is untied)")
vocab  32,000: embedding table     131M parameters (x2 if the output layer is untied)
vocab 100,000: embedding table     410M parameters (x2 if the output layer is untied)
vocab 256,000: embedding table   1,049M parameters (x2 if the output layer is untied)

For a model with 4,096-dimensional embeddings, a 256,000-token vocabulary costs about a billion parameters in the input table alone. In a model with tens of billions of parameters that is affordable; in a model of 124 million (GPT-2 small, whose embeddings already make up nearly a third of its parameters) it would be absurd. This is why vocabularies have grown with models. As examples: GPT-2 used 50,257 tokens; Meta’s Llama 2 used 32,000; Llama 3 moved to about 128,000; OpenAI’s GPT-4o tokeniser has about 200,000; Google’s Gemma models use about 256,000 As of 2026. Larger vocabularies have also been motivated by better coverage of non-English languages and code.

Quirks you can now explain

Many puzzling behaviours of language models trace back to tokenisation. Type into the box below to get a feel for how text breaks into pieces.

This box is an illustration only: it uses a simple rule (split on spaces and punctuation, chop long words into chunks), not a trained BPE vocabulary. Real tokenisers learn their pieces from data, so exact splits and counts differ between models. To see a real tokeniser at work, try a provider’s online token counter or the open-source tiktoken or Hugging Face tokenizers libraries.

Why did models struggle to count letters?

Because “strawberry” arrives as one to three tokens, not ten letters. The model learns a vector per token; whatever it knows about spelling must be inferred indirectly from training data. Newer models do much better, partly by reasoning step by step and spelling words out, but the underlying constraint is still there.

Why was arithmetic on long numbers unreliable?

Numbers tokenise irregularly. A merge-based tokeniser might split 1234567 as “123”, “456”, “7” in one tokeniser and quite differently in another, and a number seen often (such as a year) may be one token while its neighbour is two. Designers responded with explicit rules: the original LLaMA models split every number into single digits, while the tokeniser used by GPT-4 groups digits into chunks of at most three As of 2026. Either way, the model sees digits in groups that do not line up with place value, which makes carrying harder to learn.

Why does a leading space matter?

In most byte-level BPE tokenisers, “ hello” (with a space) and “hello” are different tokens with different embeddings, because the space is glued to the start of the word. A prompt that ends in a trailing space can therefore push the model into an unusual state: it must predict a word that does not start with a space, which is rarer in training. Our _ marker plays the same role.

Does Bahasa Melayu cost more tokens than English?

Usually somewhat more, for the same content, with tokenisers trained mostly on English: Malay is written in the Latin alphabet, so it is cheaper than many scripts, but its affixed words (memperkenalkan, keberkesanan) are less likely to exist as whole tokens. Scripts such as Tamil, Thai or Burmese can cost several times more. Petrov and colleagues (2023) found that translations of the same text could differ in token count by up to 15 times across languages. Since API pricing and context windows are measured in tokens, this is a real cost and fairness issue; see Working with Context and Calling LLM APIs.

Can a token be “glitchy”?

Yes. If a string was frequent in the tokeniser’s training data but rare in the model’s training data, it gets its own token whose embedding is barely trained. Researchers found such under-trained tokens in several models and showed that prompting with them could produce bizarre output. The fix is to build the tokeniser from data that resembles the pretraining data.

From token ids to vectors

The tokeniser’s output is a list of integers. The model’s first layer turns each integer into a vector — its embedding — by looking up a row of a learned matrix, the embedding table E, with one row per vocabulary entry and dmodel columns:

et = E[idt] = onehot(idt) · Ewhere E has shape (vocabulary size × dmodel), idt is the integer for token t, and onehot(idt) is a vector of zeros with a single 1 at position idt
import numpy as np

rng = np.random.default_rng(0)
vocab = ["_saya", "_makan", "_nasi", "_lemak", "an", "."]
d = 4
E = rng.standard_normal((len(vocab), d)).round(2)   # the embedding table

ids = [0, 1, 2, 3]                                  # "saya makan nasi lemak"
print("lookup:\n", E[ids])

one_hot = np.eye(len(vocab))[ids]                   # (4, 6) rows of 0s and a 1
print("same as one-hot @ E:", np.allclose(one_hot @ E, E[ids]))
print("table shape:", E.shape, " output shape:", E[ids].shape)
lookup:
 [[ 0.13 -0.13  0.64  0.1 ]
 [-0.54  0.36  1.3   0.95]
 [-0.7  -1.27 -0.62  0.04]
 [-2.33 -0.22 -1.25 -0.73]]
same as one-hot @ E: True
table shape: (6, 4)  output shape: (4, 4)

Indexing with E[ids] returns one row per token, and multiplying a one-hot matrix by E gives exactly the same thing. That equivalence matters: it shows the embedding is an ordinary linear layer, trained by backpropagation like every other weight. Only the rows of tokens that appear in a batch receive gradient, which is why rare tokens learn slowly. Libraries implement the lookup as indexing, never as a real matrix multiplication, because multiplying by a vector of zeros is wasteful.

At initialisation the rows are random, as above. After training, tokens used in similar contexts end up with similar vectors — the distributional idea behind word2vec (Mikolov and colleagues, 2013), now learned as a side effect of next-token prediction. Many models reuse the same matrix at the other end: the final hidden vector is multiplied by Eᵀ to give a logit for every token (“weight tying”), as in GPT-2 and the toy forward pass on Transformers in Depth.

Two kinds of embedding

Do not confuse these token embeddings with the sentence embeddings you met in Embeddings and Semantic Search. A token embedding is one row of a table at the bottom of a language model, context-free: “bank” has the same input row in “river bank” and “bank account”. A sentence embedding from an embedding API is the output of a whole network, after attention has mixed in context, pooled into one vector for search. The Transformer’s job is precisely to turn the first kind into context-rich vectors.

Quick check

In our BPE run, why was a + n the first merge?

BPE is pure counting: “an” appeared ten times, more than any other pair.

Why can a byte-level BPE tokeniser encode any string?

The 256 byte tokens are the floor; merges only add longer pieces on top.

What is the main cost of a much larger vocabulary?

Larger vocabularies shorten sequences but grow the tables linearly (about a billion parameters at 256,000 × 4,096) and spread training data thinner per token.

// Pause and reflect

  1. If you were building a tokeniser for a model serving Malaysian users, what would you put in its training corpus, and in what proportions?
  2. Our BPE learned _makan + an without knowing any grammar. Is that evidence that frequency captures morphology, or a lucky coincidence of a small corpus?
  3. Token-based pricing means the same request can cost more in one language than another. Whose problem is that to fix — tokeniser designers, providers or customers?

Your notes are saved privately in this browser.

Try this

Paste a few paragraphs of your own writing — one in English, one in Bahasa Melayu, on similar topics — into the BPE script’s text, raise the number of merges to 100, and compare how many tokens each paragraph needs afterwards. Then switch the script to byte-level: replace tuple("_" + w) with tuple(bytes(" " + w, "utf-8")) and make merge join pairs into tuples instead of strings. Which language benefits most from the extra merges?

Key terms on this page

Sources & further reading