Distillation
A technique where a smaller 'student' model learns to replicate the behavior of a larger 'teacher' model. Produces compact models that retain most of the teacher's capability while being faster, cheaper, and deployable on smaller devices.
Why it matters
Big models are accurate but slow and expensive to run. Distillation lets companies capture most of that quality in a model small enough to run on a phone, a laptop, or a cheap server. That's why you can now get useful AI responses instantly and often for free. Many of the fast, low-cost models you compare on aggregators started as distilled versions of a heavyweight teacher.
A concrete example
Suppose a company has a giant model that answers customer questions brilliantly but costs too much to run at scale. They train a small student model on the big model's answers to thousands of questions. The student learns to mimic those responses closely. Now it handles routine support tickets at a fraction of the cost, and the expensive teacher is reserved only for the trickiest cases.
How to use it
Distilled models are why a small, fast, cheap model can be surprisingly capable at a narrow task. When choosing, this is the reason to test rather than assume: a distilled small model can outperform a larger general one on the work it was distilled for, while being clearly worse outside it. If you are running your own, distillation is the standard route to a deployable size. The idea goes back to Distilling the Knowledge in a Neural Network.
The common mistake
Expecting the small model to inherit the large one's range. It inherits the behaviour it was trained to copy, so it holds up on the target task and degrades faster than expected on anything adjacent.
Related terms
Quantization
A technique that reduces AI model size and speeds up inference by representing model weights with fewer bits (e.g., 4-bit instead of 32-bit). Enables running large models on smaller devices with minimal quality loss.
Fine-Tuning
The process of further training a pre-trained AI model on a specific dataset to improve performance for a particular task or domain.
Inference Cost
The computational expense of running a trained AI model to generate outputs. Measured in cost per token or per request. Varies dramatically between models — GPT-4 class models cost 10-50x more per token than smaller models.
AI Dubbing
Automated translation and re-voicing of audio/video content into other languages while preserving the original speaker's voice characteristics, timing, and emotional delivery.
AI Orchestration
Coordinating multiple AI models, tools, and data sources in a unified pipeline. An orchestration layer manages prompt routing, context passing, error handling, and output aggregation across different AI services.
API (Application Programming Interface)
A way for software applications to communicate with each other. AI APIs let developers integrate AI capabilities into their own applications programmatically.