Tokenization is how text becomes numbers a model can process. A tokenizer splits input text into tokens — usually subword pieces — and maps each to an integer ID. Modern LLMs use tokenizers like Byte Pair Encoding (BPE) or SentencePiece.
One English word averages roughly 1.3 tokens. Non-English languages, code, and unusual characters produce more tokens. This matters because LLM pricing is per token and the context window is measured in tokens.
Practical consequences: "Hello world" is 2 tokens, but "antidisestablishmentarianism" might be 6. Chinese, Arabic, and code-heavy text can cost 2-4× more tokens than plain English for the same information. Nagro Solutions accounts for this in cost modelling for every LLM feature we ship.