Model Distillation: Compressing Intelligence Without Losing It

The biggest models in the world are turning into teachers for the smaller ones.

What distillation is

Knowledge distillation trains a small "student" model to mimic a large "teacher" model. Instead of training the student from scratch on raw data, you use the teacher's outputs as the training signal. The student doesn't only pick up the right answers — it picks up the teacher's uncertainty, its preferences, and the way it weighs alternatives.

The idea is simple enough. A teacher's output distribution over possible tokens carries far more information than a hard label does. When a teacher predicts "Paris" for "The capital of France is ___" with 92% probability, it's also saying "Lyon" got 3%, "Marseille" got 1%, and "Berlin" got 0.4%. Those numbers encode relationships between concepts that a one-hot label throws away.

That's a different beast from training on a dataset of correct answers. The dataset says "Paris." The teacher says "almost certainly Paris, and here's how confident I am about every alternative."

Hinton's original framework

Geoffrey Hinton's 2015 paper "Distilling the Knowledge in a Neural Network" formalized the idea. The mechanism is temperature scaling on the softmax output — a temperature $T$ in the denominator of the exponent:

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

# Standard softmax (temperature = 1)
softmax(logits)  # Sharp distribution — mostly probability on top-1

# Softened softmax (temperature = T, e.g. T=4)
softmax(logits / T)  # Flatter distribution — reveals teacher's "dark knowledge"

At higher temperatures the distribution flattens, and you start to see the teacher's soft preferences among lower-ranked outputs. Hinton called this "dark knowledge" — information about how the data is structured that lives inside the teacher but never makes it into hard labels.

The student's training loss combines two terms:

loss = alpha * cross_entropy(student_output, hard_labels) \
     + (1 - alpha) * kl_divergence(
         softmax(student_logits / T),
         softmax(teacher_logits / T)
       ) * (T ** 2)

Written out, with $p_S^{T}$ and $p_T^{T}$ the temperature-softened student and teacher distributions:

\mathcal{L} = \alpha \, \mathcal{L}_{\text{CE}}(y,\, p_S) \;+\; (1-\alpha)\, T^2 \, \mathrm{KL}\!\left(p_T^{T} \,\|\, p_S^{T}\right)

The first term keeps the student grounded in the actual correct answers. The second term transfers the teacher's distributional knowledge. The $T^2$ factor compensates for the gradient shrinking at higher temperatures — softening by $T$ scales the gradients by $1/T^2$ , so multiplying the loss back by $T^2$ keeps the two terms balanced.

Modern LLM distillation

Distillation for large language models has moved well past Hinton's original setup. Three approaches show up in practice today.

Output distillation

The student is trained on the teacher's token-level probability distributions. For each input, the teacher emits a full distribution over the vocabulary at each position, and the student learns to match it.

This is the cleanest approach, but it requires access to the teacher's logits, which you don't always have. API-only models like GPT-4 or Claude don't expose their probabilities. When logits aren't available you can still distill from generated text, but you lose the soft label information that makes the technique effective in the first place.

Feature distillation

The student learns to match the teacher's internal representations: hidden states from intermediate layers, attention patterns, or other activations. This carries richer information than output-only distillation because intermediate representations capture how the model processes information, not just what it concludes.

# Simplified feature distillation
for layer_s, layer_t in zip(student.layers, teacher.selected_layers):
    # Project student features to teacher's dimension if needed
    projected = projection_layer(layer_s.output)
    feature_loss += mse_loss(projected, layer_t.output)

You need architectural compatibility (or projection layers) and white-box access to the teacher. In practice this is typically done when both models come from the same family.

Sequence-level distillation

The teacher generates complete reasoning traces or chain-of-thought sequences, and the student is fine-tuned on those outputs. The student doesn't learn the teacher's probability distribution — it learns to reproduce the teacher's reasoning process.

This is the most practically accessible form because it only needs the teacher's text output, not its internal states. It's also the approach behind some of the most impressive recent results.

Why it works so well

The effectiveness comes down to the information density of soft labels versus hard labels.

Take a sentiment classification task with a subtle example. The hard label says "positive." But the teacher's soft distribution says "73% positive, 22% neutral, 5% negative." That 22% neutral signal tells the student this is a borderline case. Train on many such cases and the student starts to learn where decision boundaries actually sit, not just which side of the boundary each example falls on.

For language models the same logic extends to every token prediction. The teacher's distribution over vocabulary tokens encodes synonyms, stylistic preferences, domain conventions, and uncertainty. A student trained on those distributions develops a more nuanced generation capability than one trained on one-hot labels from a dataset.

Real-world results

The evidence that distillation works at scale is hard to argue with.

Microsoft's Phi-3-mini (3.8B parameters) matches or beats GPT-3.5 Turbo on many benchmarks. That wasn't pure distillation — synthetic data generation by larger models and careful data curation did a lot of the heavy lifting — but the principle is the same. Small model trained on high-quality signal from larger models.

DeepSeek R1 Distill variants are maybe the cleanest example. DeepSeek took their R1 reasoning model and distilled it into smaller models at several sizes:

DeepSeek-R1-Distill-Qwen-1.5B
DeepSeek-R1-Distill-Qwen-7B
DeepSeek-R1-Distill-Qwen-14B
DeepSeek-R1-Distill-Qwen-32B
DeepSeek-R1-Distill-Llama-8B
DeepSeek-R1-Distill-Llama-70B

The 32B distilled variant runs remarkably close to the full R1 model on reasoning benchmarks. The distillation specifically preserved the chain-of-thought behavior — the student learned not just the teacher's answers but its step-by-step reasoning process.

Google's Gemma models lean on distillation from larger Gemini models, which is a big part of why they punch above their weight class.

The limitations

Distillation isn't magic.

The student can't exceed the teacher. A distilled model's ceiling is the teacher's capability. If the teacher gets a question wrong, the student will probably get it wrong too — or worse, get it wrong in a correlated way that's harder to detect.

Quality depends on data coverage. Distillation works best when the training data covers the distribution the student will see in production. A student distilled on English text won't handle Japanese well, even if the teacher could.

Complex reasoning compresses lossy. Simple factual tasks distill cleanly. Multi-step reasoning, mathematical problem-solving, and tasks that need deep world knowledge lose more during compression. The student learns the shape of the reasoning but doesn't always develop the depth of the teacher.

Evaluation is tricky. Benchmark performance can mislead you. A distilled model might match the teacher on standard benchmarks and then fall apart on edge cases or distribution shifts the teacher handles fine. Production evaluation on your actual use case is non-negotiable.

The production angle

Distilled models are the workhorses of deployed AI. The economics are blunt:

Metric	Teacher (70B)	Distilled student (7B)
Inference cost	~10x	~1x
Latency (TTFT)	~2s	~200ms
Memory	~140GB FP16	~14GB FP16
Quality	100% (baseline)	85-95% on most tasks

For most production use cases, 85-95% quality at 10% of the cost is the right trade-off. You deploy the distilled model for the majority of requests and route only the hardest queries to the larger model.

# Production pattern: routing by complexity
if estimated_complexity(query) > THRESHOLD:
    response = large_model.generate(query)  # 5% of traffic
else:
    response = distilled_model.generate(query)  # 95% of traffic

The future of AI deployment isn't running the biggest model possible. It's running the smallest model that's good enough for each task, and distillation is how you build those models.