Token
The basic unit of text that AI models process — roughly 3/4 of a word in English. 'Unbelievable' is 3 tokens. Token limits determine how much text a model can process at once.
Why it matters
Tokens are how AI usage is measured and priced, and how limits are enforced. This matters practically: pasting a huge document can exceed a model's capacity or run up costs, and "the AI forgot the start of our chat" usually means you hit a token limit. Knowing tokens exist helps you understand billing on API tools and why very long inputs sometimes get truncated.
In practice
The word "unbelievable" might split into pieces like "un," "believ," and "able," so it counts as several tokens, not one. A rough rule of thumb: 1,000 tokens is about 750 English words. So if a tool allows an 8,000-token input, you can fit roughly 6,000 words, a longish article, but not an entire book, before you'd need to trim or split it.
Related terms
Artificial Intelligence (AI)
The simulation of human intelligence by computer systems, including learning, reasoning, and self-correction. Modern AI is primarily powered by machine learning and neural networks.
Context Window
The maximum amount of text (measured in tokens) an AI model can consider at once. Larger context windows allow the model to reference more information in a single conversation.
Large Language Model (LLM)
An AI model trained on vast amounts of text data that can generate, summarize, translate, and analyze human language. Examples include GPT-4, Claude, Gemini, and Llama.
Natural Language Processing (NLP)
The branch of AI focused on enabling computers to understand, interpret, and generate human language. Powers chatbots, translation, sentiment analysis, and text summarization.
Prompt
The text instruction you give to an AI model to generate a response. Prompt quality directly impacts output quality — better prompts yield dramatically better results.
Tokenizer
The component that splits text into tokens (sub-word units) before feeding it to an AI model. Different models use different tokenizers — affecting how they count input length, handle multilingual text, and process code.