Skip to content
AImpact
IT EN
Training Intermediate Also known as: Distillazione

Distillation

A technique to train a small model to mimic the behavior of a large one, getting similar quality at a fraction of inference cost.

ShareLinkedInX

In practice

It is why we keep getting small but capable models: they are distilled from frontier ones. If you need fast cheap responses in a narrow domain, distilling your own model from Claude or GPT is often the winning move.

Related terms

Fine-tuning Quantization

Seen in the wild

3 entries mentioning it
  1. s1: 1000 examples and a prompt trick to replicate a reasoning model
    High
  2. Gemma 2: Google's second-gen open model with Gemini distillation
    High
  3. Latent Consistency Models: real-time image generation in 4 steps
    High
← All terms