Inference Cost
The computational expense of running a trained AI model to generate outputs. Measured in cost per token or per request. Varies dramatically between models — GPT-4 class models cost 10-50x more per token than smaller models.
Why it matters
Training a model is a one-time expense, but inference, the cost of actually running it to answer prompts, is paid every single time someone uses it. For any real product, this is the bill that grows with popularity. It's why teams care so much about model size, caching, and routing, and why the price gap between a small model and a top-tier one can be enormous for the same task.
A concrete example
A startup launches a chatbot and it goes viral. Each message runs the model, and those cents per request pile up into thousands of dollars a day. To keep the doors open, they route easy questions to a cheaper model and reserve the expensive one for hard cases. Comparing per-token prices across options on an aggregator like Vincony helps them pick the most cost-effective model for each job.
How to use it
Meter it from the first day of anything that runs in a loop. The three levers, in order of effect: use a smaller model where the task is mechanical, send less input, and ask for shorter output. Cache anything repeated. And set a hard cap at the provider — the expensive failure is never one costly request, it is a loop nobody noticed running overnight.
The common mistake
Pricing a product like software while costing it like usage. Software margins assume the next user is nearly free; per-request model costs do not, so a flat price that works at ten users can lose money at a thousand.
Related terms
Token
The basic unit of text that AI models process — roughly 3/4 of a word in English. 'Unbelievable' is 3 tokens. Token limits determine how much text a model can process at once.
Model Routing
Automatically selecting the best AI model for each query based on task type, complexity, cost, and latency requirements. Smart routers analyze your prompt and route it to the optimal model — saving money on simple tasks and ensuring quality on complex ones.
Prompt Caching
A performance optimization that stores and reuses the processed representation of repeated prompt prefixes. Reduces latency and cost when sending similar prompts — especially useful for system prompts and few-shot examples.
Agentic AI
AI systems that operate autonomously over extended tasks — planning, executing, and self-correcting without step-by-step human guidance. Unlike chatbots, agentic AI sets sub-goals, uses tools, and adapts its strategy based on intermediate results.
AI Agent
An autonomous AI system that can perceive its environment, make decisions, and take actions to achieve goals — like managing your email, scheduling meetings, or monitoring data.
AI Disclosure
Stating that AI was used in producing a piece of work, where a policy, client or publisher requires it. Distinct from permission: some contexts allow AI use but require it to be declared.