AI Model Distillation Explained

Model distillation trains a small "student" neural network to reproduce the behavior of a much larger "teacher" model, using the teacher's own predictions as the training signal instead of raw labels. The technique was formalized by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in a 2015 paper, and it now underpins some of the most widely used AI systems in production, from Hugging Face's DistilBERT to Google's Gemma and Anthropic's smaller Claude models. Here is how it actually works, why it became the default way to make AI cheaper to run, and why it triggered a legal fight between OpenAI and DeepSeek.

What distillation actually is

A large language model with hundreds of billions of parameters is expensive to run. Every query means loading billions of numbers into GPU memory and running trillions of matrix multiplications, which costs money, electricity, and time. Distillation is the standard fix: train a much smaller model, sometimes a tenth the size or less, to copy the larger model's behavior closely enough that most users cannot tell the difference in everyday use.

The "teacher" is the large, already-trained model. The "student" is the smaller model being trained. Instead of training the student from scratch on the same raw data the teacher used, engineers feed the same inputs to both models and train the student to match the teacher's output distribution, not just the final answer. That distinction is the entire trick, and it is worth unpacking carefully because it is where most people misunderstand the technique.

Think of it like an apprenticeship. A textbook (raw labeled data) tells you the correct answer to a problem and nothing else. A skilled mentor (the teacher model) instead shows you how confident to be about each possible answer, which near-misses were plausible, and which were absurd. A student who learns from the mentor's full reasoning picks up patterns that a "correct answer only" textbook could never convey.

DistilBERT, released by Hugging Face in 2019, cut BERT's parameter count by 40 percent and ran 60 percent faster while retaining 97 percent of BERT's language understanding performance on the GLUE benchmark, based on distillation alone with no architecture redesign (arXiv:1910.01108).

How the teacher-student training actually works

The mechanism at the heart of distillation is what Hinton's paper called "soft targets," and it depends on a detail of how neural networks make predictions that most explanations skip over.

What are soft targets in knowledge distillation?

A classifier's final layer produces a softmax probability distribution across possible answers, like 92 percent "cat," 6 percent "dog," 2 percent everything else. Hard labels used in normal training just say "cat" and discard the rest. Soft targets keep the whole distribution, including the small probability assigned to "dog," because that residual signal tells the student which classes are relatively similar to each other and which are unrelated.

Hinton's paper introduced a parameter called temperature to control how soft those targets are. The softmax formula divides each logit (the model's raw pre-probability score) by a temperature value T before normalizing. At T equal to 1, you get the model's normal, fairly confident output. Raise T to 10 or 20, and the distribution flattens out, exposing small differences between wrong answers that a temperature of 1 would round down to nearly zero. In Hinton's MNIST digit-recognition experiments, a small student network trained purely by matching a large network's soft targets at a temperature of 20 dropped to 74 test errors, a result the paper used to demonstrate that soft targets alone carry enough signal to train a competent student (Hinton, Vinyals, and Dean, 2015).

Why does the student model end up smaller but almost as accurate?

Because the soft-target signal is denser than a hard label, the student needs far fewer parameters to extract the same amount of useful structure from each training example. A hard label is one bit of information per example. A full probability distribution over, say, 50,000 possible next tokens is thousands of times richer, so a smaller network can converge on comparable accuracy in fewer training steps with fewer weights to store.

Distillation loss: how do you actually train the student network?

In practice, engineers combine two loss terms. One measures how far the student's soft predictions are from the teacher's soft predictions (usually with Kullback-Leibler divergence). The other is a normal supervised loss against the ground-truth hard labels, when those exist. DistilBERT's authors added a third term, a cosine-distance loss between the teacher's and student's hidden-state vectors, which pushed the student to mimic not just the teacher's final answer but its internal representations of language (Sanh et al., 2019, arXiv:1910.01108). Weighting those loss terms correctly is most of the practical engineering work in a distillation project.

The main types of distillation used in production

Distillation is not one single recipe. Depending on what a team needs, engineers reach for a handful of distinct variants that differ in what part of the teacher they copy and when.

Distillation type	What the student learns from	Where it's used
Response-based (soft-label) distillation	The teacher's final output probabilities	DistilBERT, most chatbot compression pipelines
Feature-based distillation	The teacher's intermediate hidden layers, not just outputs	DistilBERT's cosine-loss term, vision model compression
Self-distillation	A model distills knowledge from an earlier or larger version of itself	Iterative model refinement, some Gemma training runs
Data distillation (synthetic-data distillation)	A large model's generated text, used as fresh training data for a smaller model	Alpaca-style instruction tuning, most open-source fine-tunes since 2023
Online distillation	Teacher and student trained together, teacher updated in parallel	Large-scale pretraining where a fixed pre-trained teacher isn't available

Table: Common distillation variants and where each is applied in real AI pipelines, based on published technical reports cited below.

Google's Gemma family illustrates the pretraining-time version well. According to the Gemma 2 technical report, the 9-billion-parameter model was pretrained using knowledge distillation from a larger 27-billion-parameter teacher, while the 27B model itself was trained from scratch on raw data, since no larger Gemma model existed to teach it. The report found that distilling from a bigger model produced meaningfully better results than training the smaller model on the same token budget without a teacher (Google DeepMind, Gemma 2 Technical Report). Gemma 3 extended this further: all Gemma 3 model sizes, from 1 billion to 27 billion parameters, were trained with distillation as a core part of pretraining rather than an afterthought applied only for deployment (Gemma Team, Google DeepMind, Gemma 3 Technical Report, 2025).

Why distillation matters for cost, latency, and access

The practical case for distillation comes down to three numbers that matter to anyone deploying a model: memory footprint, inference latency, and dollar cost per query. A smaller model needs less GPU memory, which means it can run on cheaper hardware or even a phone. It also processes each request faster, since there are fewer matrix multiplications to perform, and processes more requests per GPU, which lowers cost per query at scale.

This is why a distilled model rarely needs to match its teacher exactly to be useful. DistilBERT's 97-percent-of-BERT accuracy at 40 percent of the size is the classic illustration: for a huge share of production text-classification workloads, a 3-percentage-point accuracy gap is a trivial cost against a 60-percent latency improvement (Hugging Face, Sanh et al., 2019). The same logic explains why most consumer-facing AI products today route simple queries to small distilled models and reserve the largest, most expensive model for genuinely hard requests.

How is distillation different from quantization or pruning?

Quantization shrinks a model by reducing the numerical precision of its existing weights, for example storing them as 8-bit integers instead of 32-bit floats, without changing the model's architecture or retraining it. Pruning removes individual weights or entire neurons that contribute little to the output. Distillation is different in kind: it trains an entirely separate, smaller network from scratch using the teacher's outputs as a training signal. The three techniques are complementary, and a production model is often distilled first, then quantized and pruned on top of that.

Can distillation work across different model architectures?

Yes. Because the student only needs to match the teacher's output distribution, the two networks do not need matching architectures. A transformer can distill knowledge into a much smaller transformer with fewer layers, as DistilBERT does with six layers versus BERT-base's twelve, or even into an entirely different architecture, as long as the student can be trained to approximate the same probability distributions.

The synthetic-data version: distillation without direct access to weights

The version of distillation causing the most controversy in 2025 and 2026 does not require access to a teacher model's internals at all. If a lab does not have direct access to a competitor's model weights, it can still distill knowledge by querying the competitor's API repeatedly, collecting the outputs, and using that generated text as training data for its own model. This is sometimes called data distillation or synthetic-data distillation, and it is functionally the same mentor-apprentice relationship, just conducted entirely through a public interface instead of internal probability logs.

This is exactly what OpenAI alleged DeepSeek did. In testimony to the US Congress and in public statements, OpenAI claimed that DeepSeek queried OpenAI's models at scale and used the responses as training data to build a competing model, which OpenAI argues violated its terms of service (Vision Times, reporting OpenAI's February 2026 congressional testimony; Berkeley Law, "The Innovation Dilemma: AI Distillation in OpenAI v. DeepSeek," March 2025). DeepSeek has denied the specific allegation of improperly extracting proprietary data (TechTakes, "It Wasn't Me," 2026).

Major AI labs, including OpenAI, Anthropic, Mistral, and xAI, now include explicit "anti-competitive distillation" clauses in their terms of service, prohibiting customers from using a model's outputs to train a competing model. This is a contractual restriction, not a technical one; nothing about the API itself prevents the technique, which is why the dispute has moved into legal rather than purely engineering territory ([law.asia, "Dispute over AI model distillation tech in OpenAI-DeepSeek case"](https://law.asia/openai-deepseek-ai-distillation/)).

Legal commentary on the OpenAI-DeepSeek dispute has generally noted that distillation from a model's public outputs sits in an unsettled area of both contract and intellectual-property law: a terms-of-service violation is a contractual claim against whoever agreed to those terms, but it is a separate and harder legal question whether the outputs themselves are protected in a way that make training on them unlawful for a party that never agreed to the terms (Winston & Strawn, "Is AI Distillation By DeepSeek IP Theft?").

What people get wrong about distillation

The most common misunderstanding is that a distilled model is just a "compressed" or "shrunk" copy of the teacher, the same way a ZIP file compresses a document without losing any information. That analogy breaks down immediately. A distilled model is a fresh network trained from a different starting point on a different signal. It does not contain the teacher's weights in any recoverable form, and it can develop its own quirks, blind spots, and even outperform the teacher on narrow tasks it was specifically distilled for.

A second common mistake is assuming that any small model advertised as fast and cheap must be a distilled version of some bigger flagship model. Plenty of small models, including many open-source releases, are trained from scratch on raw data rather than distilled from a teacher. What small language models are good for covers where models built this way, rather than through distillation, tend to earn their keep, and the two approaches are not mutually exclusive.

A third mistake is treating "97 percent of teacher performance" figures as if they apply uniformly across every task. DistilBERT's 97-percent figure comes from averaged GLUE benchmark scores. On individual tasks, the gap between teacher and student can be considerably wider or narrower, and benchmark averages hide that variance. Why AI benchmarks can be misleading goes into how averaged scores like this can obscure exactly the kind of task-specific weaknesses that matter most in production.

Finally, people sometimes assume distillation only happens once, at the end of a model's development, as a compression step tacked onto a finished product. Gemma 2 and Gemma 3's technical reports show the opposite is increasingly common: distillation built directly into the pretraining process from the start, not as an afterthought but as the primary training method for every model size below the largest one in a family.

Why distillation depends on how transformers represent knowledge in the first place

Distillation only works because of a specific property of how modern language models generate text: at every step, they produce a full probability distribution over the entire vocabulary, not a single deterministic answer. That distribution exists because of the self-attention mechanism inside a transformer, which weighs how relevant every other token in the input is to the one being predicted. How transformers pay attention to words explains that mechanism in detail, and it is worth understanding before diving deeper into distillation, because the richness of that attention-weighted probability distribution is precisely the signal a teacher model hands down to a student.

It also explains why distillation is such an effective lever against the resource problem described in why AI needs so much computing power: most of the FLOPs (floating-point operations) consumed by a large model at inference time go toward computing attention and matrix multiplications across billions of parameters that, for the vast majority of everyday queries, are not doing much useful work. Distillation is one of the few techniques that directly attacks that inefficiency rather than working around it with better hardware.

The takeaway

Model distillation turns an expensive, hard-to-run AI model into a cheaper one by training a smaller network to copy the original's full probability distributions rather than just its final answers, a technique formalized by Hinton, Vinyals, and Dean in 2015 and now standard in products from DistilBERT to Gemma. The core trade-off is real but often small: DistilBERT keeps 97 percent of BERT's accuracy at 40 percent of the size, and Google now builds distillation directly into how it pretrains every Gemma model below its largest. The technique's newest wrinkle is legal rather than technical: querying a competitor's model at scale to distill it into your own now sits at the center of an unresolved dispute between OpenAI and DeepSeek, over contracts and intellectual property that predate anyone anticipating this use case.

---

Read next:

• How Transformers Pay Attention To Words
• What Small Language Models Are Good For
• Why AI Needs So Much Computing Power

Frequently Asked Questions

Q: What is AI model distillation in simple terms? A: Model distillation trains a smaller "student" neural network to imitate a larger "teacher" model's outputs, including the teacher's full probability distributions rather than just its final answers, which lets the student reach much of the teacher's performance at a fraction of the size and cost.

Q: Who invented knowledge distillation? A: Geoffrey Hinton, Oriol Vinyals, and Jeff Dean formalized the technique in their 2015 paper "Distilling the Knowledge in a Neural Network," introducing the concept of "soft targets" and a temperature parameter to control how much information the teacher's output distribution reveals to the student.

Q: What is a soft target in distillation, and why does it matter? A: A soft target is the full probability distribution a teacher model assigns across all possible answers, not just the top prediction. It carries far more information per training example than a single "correct" label, which is why a student trained on soft targets can reach strong accuracy with fewer parameters than one trained from scratch.

Q: Is DistilBERT actually smaller than BERT, and by how much? A: Yes. DistilBERT has roughly 66 million parameters versus BERT-base's 110 million, a 40 percent reduction, runs about 60 percent faster, and retains about 97 percent of BERT's performance on the GLUE language-understanding benchmark, according to Hugging Face's 2019 paper.

Q: How is distillation different from quantization? A: Quantization reduces the numerical precision of an existing model's weights without retraining or changing its architecture. Distillation trains an entirely new, typically smaller network from scratch using the teacher's outputs as the training signal. The two techniques are often combined, with a model distilled first and quantized afterward.

Q: Why did OpenAI accuse DeepSeek of illegal distillation? A: OpenAI told the US Congress in testimony reported in February 2026 that DeepSeek queried OpenAI's models at scale and used the responses as training data for its own models, which OpenAI says violated its terms of service. DeepSeek has denied improperly extracting proprietary data, and the dispute remains unresolved as of this writing.

Q: Can a distilled model ever outperform its teacher? A: On narrow tasks it was specifically trained for, yes, a distilled student can match or exceed its teacher, since the student's smaller size can make it less prone to overfitting on noise in that narrow task. It typically cannot exceed the teacher's broad, general-purpose capability, since it never sees more information than the teacher provides.

Q: Do all small AI models use distillation? A: No. Many small language models are trained from scratch on raw datasets rather than distilled from a larger teacher. Distillation is one method among several, including pruning, quantization, and training from scratch, for producing a smaller, cheaper model, and production pipelines frequently combine more than one of these techniques.

Q: Does distillation require access to the teacher model's internal weights? A: Not always. Response-based and feature-based distillation typically require access to the teacher's internal logits or hidden layers. But a variant called data or synthetic-data distillation only requires querying the teacher's public API and training on its outputs as if they were regular text data, which is the method at the center of the OpenAI-DeepSeek dispute.

Q: Why do companies put anti-distillation clauses in their terms of service? A: Because distillation lets a competitor build a comparably capable model far more cheaply than training one from scratch, using another company's already-trained model as a shortcut. Labs including OpenAI, Anthropic, Mistral, and xAI now prohibit this contractually, since there is no purely technical way to prevent a paying customer from training on a model's public outputs.

Sources

• Hinton, G., Vinyals, O., and Dean, J. (2015). "Distilling the Knowledge in a Neural Network."
• Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." arXiv:1910.01108.
• Hugging Face. "Smaller, faster, cheaper, lighter: Introducing DistilBERT."
• Google DeepMind. "Gemma 2 Technical Report" / Gemma explained.
• Gemma Team, Google DeepMind (2025). "Gemma 3 Technical Report."
• Vision Times (2026). "OpenAI Warns Congress That DeepSeek Is Illegally 'Distilling' US AI Models."
• Berkeley Law, The Network (2025). "The Innovation Dilemma: AI Distillation in OpenAI v. DeepSeek."
• law.asia. "Dispute over AI model distillation tech in OpenAI-DeepSeek case."
• Winston & Strawn. "Is AI Distillation By DeepSeek IP Theft?"