LLM Tokenization Explained

Language models do not read text the way humans do. Before any processing happens, text must be converted into a sequence of numbers. Tokenization is the process that breaks raw text into chunks (tokens), each mapped to an integer ID. The choice of tokenizer affects everything from model performance to API costs.

Why not just use characters or words?

The simplest approach would be to split on characters. English has roughly 100 common characters, so the vocabulary is tiny. But character sequences are long, and the model has to learn relationships across many positions to understand even simple words.

Word-level tokenization is the other extreme. Each word gets its own ID. But this creates a massive vocabulary (English has hundreds of thousands of words), and any word not in the vocabulary is unrepresentable. Typos, compound words, and morphological variations all become problems.

Subword tokenization sits in the middle. It splits text into pieces that are sometimes whole words and sometimes fragments. Common words like “the” get their own token. Rare words like “tokenization” might be split into “token” + “ization”. This keeps the vocabulary manageable (32K-100K tokens) while handling any input.

Byte Pair Encoding (BPE)

BPE is the most common tokenization algorithm used by modern LLMs. It was originally a data compression algorithm adapted for NLP.

How BPE training works

1. Start with a base vocabulary of individual bytes (256 entries).
2. Scan the training corpus and count every adjacent pair of tokens.
3. Merge the most frequent pair into a new token.
4. Repeat steps 2-3 until the vocabulary reaches the target size.

# simplified BPE training
def train_bpe(text, vocab_size):
    # start with individual characters
    tokens = list(text)
    vocab = set(tokens)
    merges = []
    
    while len(vocab) < vocab_size:
        # count all adjacent pairs
        pair_counts = {}
        for i in range(len(tokens) - 1):
            pair = (tokens[i], tokens[i + 1])
            pair_counts[pair] = pair_counts.get(pair, 0) + 1
        
        if not pair_counts:
            break
        
        # find the most frequent pair
        best_pair = max(pair_counts, key=pair_counts.get)
        merges.append(best_pair)
        
        # merge that pair everywhere
        new_token = best_pair[0] + best_pair[1]
        vocab.add(new_token)
        
        new_tokens = []
        i = 0
        while i < len(tokens):
            if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == best_pair:
                new_tokens.append(new_token)
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1
        tokens = new_tokens
    
    return vocab, merges

BPE encoding

Once trained, encoding new text applies the learned merges in order:

def encode_bpe(text, merges):
    tokens = list(text)
    
    for left, right in merges:
        new_tokens = []
        i = 0
        while i < len(tokens):
            if i < len(tokens) - 1 and tokens[i] == left and tokens[i + 1] == right:
                new_tokens.append(left + right)
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1
        tokens = new_tokens
    
    return tokens

# example
text = "low lower lowest"
vocab, merges = train_bpe(text * 100, vocab_size=270)
print(encode_bpe("lower", merges))
# might output: ['low', 'er']

SentencePiece

SentencePiece, developed by Google, is a language-independent tokenizer that works directly on raw text without pre-tokenization. Models like LLaMA and T5 use SentencePiece.

Key differences from standard BPE:

• Treats the input as a raw byte stream, so it works with any language without preprocessing.
• Uses a unigram language model as an alternative to BPE (though it supports BPE too).
• Represents spaces as a special character (the “sentencepiece” underscore _).

import sentencepiece as spm

# train a SentencePiece model
spm.SentencePieceTrainer.train(
    input="training_text.txt",
    model_prefix="my_tokenizer",
    vocab_size=32000,
    model_type="bpe",          # or "unigram"
    character_coverage=0.9995,  # for CJK/multilingual
    pad_id=3,
)

# load and use
sp = spm.SentencePieceProcessor()
sp.load("my_tokenizer.model")

text = "Tokenization is fundamental to LLMs."
tokens = sp.encode_as_pieces(text)
ids = sp.encode_as_ids(text)

print(f"Tokens: {tokens}")
# ['_Token', 'ization', '_is', '_fundamental', '_to', '_LL', 'Ms', '.']
print(f"IDs: {ids}")
# [1234, 5678, 42, 8901, 15, 2345, 678, 9]

# decode back
print(sp.decode(ids))
# "Tokenization is fundamental to LLMs."

tiktoken

tiktoken is OpenAI’s fast BPE tokenizer used by GPT models. It is written in Rust with Python bindings.

import tiktoken

# get the tokenizer for a specific model
enc = tiktoken.encoding_for_model("gpt-4o")
# or by encoding name directly
enc = tiktoken.get_encoding("o200k_base")

text = "Hello, how are you doing today?"
tokens = enc.encode(text)
print(f"Token IDs: {tokens}")
print(f"Token count: {len(tokens)}")

# decode individual tokens to see the splits
for token_id in tokens:
    print(f"  {token_id} -> '{enc.decode([token_id])}'")

# decode back to text
print(enc.decode(tokens))

Comparing tokenizers across models

Different models use different tokenizers with different vocabulary sizes:

import tiktoken

models = {
    "gpt-4o": "o200k_base",
    "gpt-4": "cl100k_base",
    "gpt-3.5-turbo": "cl100k_base",
}

text = "The quick brown fox jumps over the lazy dog."

for model, encoding in models.items():
    enc = tiktoken.get_encoding(encoding)
    tokens = enc.encode(text)
    print(f"{model} ({encoding}): {len(tokens)} tokens")

How tokenization affects LLM behavior

Token limits and context windows

Every model has a maximum context length measured in tokens, not characters or words. A rough rule of thumb for English text: 1 token is approximately 4 characters or 0.75 words.

def estimate_tokens(text, model="gpt-4o"):
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

# check if a prompt fits in context
prompt = "..." # your prompt
token_count = estimate_tokens(prompt)
max_tokens = 128000  # gpt-4o context window

if token_count > max_tokens:
    print(f"Prompt too long: {token_count} tokens (max {max_tokens})")
else:
    print(f"Fits: {token_count}/{max_tokens} tokens ({token_count/max_tokens:.1%} used)")

Cost implications

API pricing is per token. Understanding how your text tokenizes helps estimate costs:

def estimate_cost(text, price_per_million_input=2.50, model="gpt-4o"):
    token_count = estimate_tokens(text, model)
    cost = (token_count / 1_000_000) * price_per_million_input
    return token_count, cost

tokens, cost = estimate_cost(long_document)
print(f"{tokens} tokens, estimated input cost: ${cost:.4f}")

Non-English text uses more tokens

Tokenizers trained primarily on English text are less efficient for other languages. The same meaning expressed in Japanese or Arabic may consume 2-3x more tokens than English.

enc = tiktoken.encoding_for_model("gpt-4o")

texts = {
    "English": "Hello, how are you?",
    "Japanese": "こんにちは、お元気ですか？",
    "Arabic": "مرحبا، كيف حالك؟",
    "Code": "def hello(): return 'hi'",
}

for lang, text in texts.items():
    tokens = enc.encode(text)
    ratio = len(tokens) / len(text)
    print(f"{lang}: {len(text)} chars -> {len(tokens)} tokens (ratio: {ratio:.2f})")

Tokenization artifacts

Some surprising behaviors in LLMs trace back to tokenization. The model does not see individual characters; it sees tokens. This means:

• Counting characters in a word is hard because the model sees token chunks, not letters.
• Reversing strings is unreliable for the same reason.
• Arithmetic can be inconsistent because numbers tokenize in unpredictable ways.

enc = tiktoken.encoding_for_model("gpt-4o")

# "123456789" might tokenize as ["123", "456", "789"] or ["1234", "5678", "9"]
number = "123456789"
tokens = enc.encode(number)
print(f"'{number}' -> {[enc.decode([t]) for t in tokens]}")

# this affects the model's ability to do digit-level operations

Building a custom tokenizer

For specialized domains, you might want to train a custom tokenizer:

from tokenizers import Tokenizer, models, trainers, pre_tokenizers

# create a BPE tokenizer
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)

trainer = trainers.BpeTrainer(
    vocab_size=32000,
    min_frequency=2,
    special_tokens=["<pad>", "<s>", "</s>", "<unk>"],
    show_progress=True,
)

# train on your domain-specific text
tokenizer.train(files=["domain_corpus.txt"], trainer=trainer)

# test it
output = tokenizer.encode("Your domain-specific text here")
print(f"Tokens: {output.tokens}")
print(f"IDs: {output.ids}")

# save for later use
tokenizer.save("domain_tokenizer.json")

Practical tips

Always count tokens, not words. When working with context limits, use the exact tokenizer for your model. Rough estimates can be off by 30% or more.

Be aware of special tokens. Chat models prepend system tokens, role markers, and delimiters. These consume part of your context window. The tiktoken library has methods to account for this.

Whitespace matters. Leading spaces, trailing newlines, and multiple spaces all produce different tokens. Normalizing whitespace before sending to an API can save tokens and improve consistency.

Code tokenizes differently from prose. Programming languages often use symbols and indentation patterns that tokenize into many small tokens. A 100-line Python function might use significantly more tokens than the equivalent explanation in English.

Tokenization is invisible most of the time, but understanding it helps you write better prompts, debug unexpected model behavior, and control costs.