Tokenizer
The component that splits text into tokens (sub-word units) before feeding it to an AI model. Different models use different tokenizers — affecting how they count input length, handle multilingual text, and process code.
Why it matters
Models don't read letters or whole words; they read tokens, the small chunks a tokenizer breaks your text into. This quietly shapes cost, since you're usually billed per token, and it explains quirks like models miscounting characters or struggling with rare words. Understanding tokens helps you write leaner prompts and makes sense of why the same sentence can cost more in one model than another.
A concrete example
The word "unbelievable" might be split into pieces like "un," "believ," and "able," while a common word like "the" is a single token. Emojis or unusual names can take several tokens each. If you paste a long document and hit a length limit, it's the token count, not the word count, that matters, which is why trimming filler text can let more of your content fit.
How to use it
The tokenizer explains a surprising amount of odd behaviour: why models miscount letters in a word, why they struggle to reverse strings or do character-level edits, and why the same text costs different amounts on different models. If a task involves manipulating individual characters, expect it to be unreliable and check the result — that failure is structural rather than a sign of a weak model.
The common mistake
Assuming one word is one token. Common words often are; longer, rarer and non-English words split into several pieces, which is why token estimates based on word counts are consistently low for anything technical or multilingual.
Related terms
Token
The basic unit of text that AI models process — roughly 3/4 of a word in English. 'Unbelievable' is 3 tokens. Token limits determine how much text a model can process at once.
Context Window
The maximum amount of text (measured in tokens) an AI model can consider at once. Larger context windows allow the model to reference more information in a single conversation.
Inference Cost
The computational expense of running a trained AI model to generate outputs. Measured in cost per token or per request. Varies dramatically between models — GPT-4 class models cost 10-50x more per token than smaller models.
Artificial Intelligence (AI)
The simulation of human intelligence by computer systems, including learning, reasoning, and self-correction. Modern AI is primarily powered by machine learning and neural networks.
Large Language Model (LLM)
An AI model trained on vast amounts of text data that can generate, summarize, translate, and analyze human language. Examples include the GPT, Claude, Gemini and Llama families.
Natural Language Processing (NLP)
The branch of AI focused on enabling computers to understand, interpret, and generate human language. Powers chatbots, translation, sentiment analysis, and text summarization.