llm from scratch, khmer tokenization cost

Work in progress. The transformer is written and the corpus is on disk. Nothing has been measured yet, so there are no numbers on this page and I would rather leave it that way than put up guesses.

Khmer is written without spaces between words. Subword tokenizers were tuned on languages that do have spaces, so they chop Khmer into far more pieces than it takes to say the same thing in English. That is not cosmetic. More tokens per sentence means a shorter effective context, a higher bill per page of text, and possibly a worse model. This project is an attempt to put a number on that, for my own language, with enough statistics behind it that the number means something.

why i built it

I started it as a way to actually understand what I was using. I was reading Raschka's Build a Large Language Model (From Scratch) and typing out the byte-pair encoder and the attention block by hand rather than cloning the repo, because I wanted to be able to change them later and not just run them.

Somewhere around the tokenization chapter I fed it Khmer. The book's tokenizer splits on whitespace before it does anything clever, and Khmer has none, so the whole first stage of the pipeline quietly does nothing useful. Every design choice in the chapter assumes a space-delimited language. That was the moment this stopped being a tutorial I was working through and became a question I wanted answered.

It is also personal in a way my other projects are not. Khmer is my language. Nobody is going to measure this for me, and the cheapest version of every language tool I use is the English one.

the thinking

The single most important decision is the metric, and it is the one most comparisons get wrong. Perplexity is not comparable across tokenizers. It is measured per token, so a tokenizer that shreds text into more, smaller pieces makes each individual prediction easier and scores a lower perplexity while doing a worse job. Compare raw perplexity across tokenizers and you get a confident, backwards answer.

The fix is to divide by something the tokenizer cannot change, the characters or bytes of the actual text. Every headline number in this project will be bits per character, with bits per byte reported alongside it, because a Khmer codepoint takes three UTF-8 bytes and the byte view is what actually drives serving cost. There is a unit test whose only job is to confirm that this number stays the same when the tokenizer changes on a fixed piece of text. If that test ever goes red the whole project is reporting nonsense.

The second decision is about what gets held still. Vocabulary size changes the embedding matrix, which changes the parameter count, so a bigger-vocab model that wins might just be a bigger model. Vocab stays fixed across tokenizer arms, embeddings are tied, and the non-embedding parameter count gets reported next to every result. Depth, width, heads, context, optimizer, schedule, seeds and token budget are all frozen. The tokenizer is the only thing that moves. That is what makes it an experiment instead of a pile of training runs.

The third is the third arm. Off-the-shelf gives me byte-pair encoding and a SentencePiece unigram model, both blind to how Khmer is actually built. Khmer orthographic syllables have regular structure over the U+1780 to U+17FF block, a base consonant, then an optional coeng plus a subscript consonant, then vowel signs, then diacritics. A regex segmenter over that pattern is a weekend of work and gives a linguistically motivated baseline neither library represents.

where the statistics come in

I am a stats student, and the part of this I am most interested in is the part that turns a table of numbers into a claim I can defend.

Training is maximum likelihood, so the loss curve is a likelihood ascent and gets reported in bits throughout. Every arm is evaluated on the same held-out documents, which makes it a paired design, so what I compare is the per-document difference rather than two condition means. That strips out the fact that some documents are simply harder than others and tightens the intervals a lot. The bootstrap resamples documents, not tokens, because tokens inside a document are heavily autocorrelated and a token-level bootstrap would badly understate the uncertainty.

The result I am most curious about is calibration. Reliability diagrams and expected calibration error over next-token predictions, with temperature fit on held-out data, and then the question I actually want answered. Is the model equally well calibrated across both languages, or is it systematically overconfident in the language whose tokenizer serves it worst? I do not know the answer. That is why it is worth running.

where it is now

Nine working days in, through chapter five of the book. The attention block, the layers and a small GPT are written out by hand, there is a training loop and a loss and generation path, and a 1.5 GB raw Khmer corpus is sitting in data/raw waiting to be cleaned, deduped and byte-matched against an English counterpart of the same size.

The next real milestone is deliberately the one that needs no GPU. Train the three tokenizers, write the syllable segmenter and its tests, and measure fertility, bytes per token, coverage and out-of-vocabulary rate across vocab sizes from 4k to 32k. That alone answers the fertility half of the question and is a reportable result whether or not the training runs ever happen on time. Training comes after, on the homelab GPU node, in overnight batches.

The thing I want to avoid is the version of this project that is a finished tutorial with a README that says what the book already says. The measurement is the point.