Part 1: Building a Byte-Pair Tokenizer

A model never sees text. It sees integers, and the mapping from one to the other is a design decision that shapes everything downstream.

Why not just use characters?

Character-level models work. They are also slow, because a sequence of 2000 characters is 2000 positions of attention to compute, and most of those positions carry almost no information.

Byte-pair encoding

The idea is simple enough to state in a sentence: repeatedly find the most common adjacent pair of tokens and merge it into a new token.

def merge(ids: list[int], pair: tuple[int, int], new_id: int) -> list[int]:
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

What to measure

Compression ratio is the number to watch: bytes in, tokens out.