Akshay Ranganath

Deep-dived into tokenization while working on my blog and here's what I learned: When I first heard “tokenization,” I thought it was just some boring preprocessing step. Like…break text into pieces, feed to model, done. But nope. It’s actually one of the biggest make-or-break design choices in LLMs. 1. What even is tokenization? Models don’t see words. They see numbers. So we need to chop up text into tokens -> turn them into IDs -> pass them in. Example: “PKMKB” might become: [“P”, “K”, “M”, “K”, “B”] And yes, “PKMKB” doesn’t always stay whole. That’s the problem. 2. Why should I care? Because bad tokenization means: Inputs become way longer than they need to be. Attention cost explodes (remember: quadratic scaling T-T) Non-English or code inputs get messy fast. Basically, tokenization quietly decides whether your GPU bill is reasonable or ouch my wallet. 3. The usual method (BPE) Most LLMs today use Byte Pair Encoding (BPE) or something similar. Common chunks of text get their own tokens (“ing”, “tion”, “Chat”). Rare stuff gets split apart. It’s simple and works decently…until you throw weird inputs at it. 4. Where it breaks Example: print("🌚") print -> 1 token ( -> 1 token 🌚 -> …FOUR tokens😭 So your little emoji costs the same as a whole word. Imagine feeding entire codebases or Hindi text, sequence length blows up. 5. New ideas People are trying learnable tokenizers -> models that adaptively group bytes into smart chunks. Instead of a fixed dictionary, it figures out better units for text, emoji, code, multiple languages. Still research-y, but feels like the future. 6. My main realization Tokenization isn’t just a footnote. It’s the hidden lever that controls compute, efficiency, and how friendly a model is to the real world (not just English essays). If Transformers are the big shiny engine… Tokenization is the fuel line. Mess it up, and the whole system chokes. 7. Final note I started this thinking “eh, tokenization is boring.” Now I think: it’s one of the most strategic choices you make when building an LLM. Tiny step, huge consequences. That’s my tokenization brain dump. Next imma dive into attention (QKV) and RLHF, stay tuned :)