Teach a smaller model

DeepSeek's 32-billion-parameter student scored 72.6% on AIME 2024, a competition mathematics benchmark. Its teacher, DeepSeek-R1, scored 79.8%. A much smaller model had learned enough to get surprisingly close on that test. For someone deciding which model to run, the gap is worth examining: what could the student do, and what would you lose by choosing it? DeepSeek-R1 report, Tables 4 and 5

That is the attraction of knowledge distillation: train a smaller "student" using a more capable "teacher" and deploy the student where it does the job. But what, exactly, did it learn? A model that can imitate a worked solution might still stumble when the problem changes. To decide whether the smaller model is useful, we need to understand what the training signal contains.

An answer can carry more than a label

Imagine a sentiment classifier reading this invented review: "The food was good. Eventually it arrived."

A training label might say "positive." That gives the student a destination, but little help with the ambiguity. Suppose a teacher instead assigns 73% to positive, 22% to neutral, and 5% to negative. These are illustrative probabilities. The 22% neutral signal tells the student that this example sits near a boundary. Across many examples, those softer judgments can help it learn how categories relate.

The same idea applies to next-token predictions. In a toy completion of "The capital of France is ___," a teacher might put 92% on "Paris," 3% on "Lyon," 1% on "Marseille," and 0.4% on "Berlin," with the remaining probability elsewhere. The incorrect alternatives carry information too: French cities receive different weights from a foreign capital. A single correct-answer label discards that pattern.

This is what a soft target provides: a distribution over possible answers. It represents the teacher's predictions, including its mistakes. Confidence alone doesn't make those predictions true.

Making the softer signal visible

Hinton, Vinyals, and Dean's 2015 distillation framework uses temperature to expose relationships among less likely outputs. For a model score, or logit, $z_i$, the probability is:

$$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$

A higher $T$ flattens the distribution. The runner-up answers become easier for the student to learn from. Distilling the Knowledge in a Neural Network

Training can combine the known correct labels with the teacher's soft targets. Here is pseudocode, with kl_divergence(p, q) defined as KL(p || q):

Written out, with $p_S^{T}$ and $p_T^{T}$ the softened student and teacher distributions:

$$\mathcal{L} = \alpha \, \mathcal{L}_{\text{CE}}(y,\, p_S) \;+\; (1-\alpha)\, T^2 \, \mathrm{KL}\!\left(p_T^{T} \,\|\, p_S^{T}\right)$$

The first term rewards correct answers. The second rewards matching the teacher's distribution. The weight $\alpha$ controls their balance. Multiplying by $T^2$ compensates for the approximate $1/T^2$ gradient scaling at high temperatures. KL argument conventions vary across libraries; an implementation must match the equation. Distillation training objective

You don't need to memorize the equation to understand the choice it makes: how much should the student trust the answer key, and how much should it imitate the teacher?

What you can transfer depends on what you can access

Token-level output distillation needs teacher probabilities for the vocabulary at each position. That is a richer signal than one generated answer, but you need access to those probabilities. A text-only interface cannot supply them, and a short list of top-token probabilities is not a full vocabulary distribution.

Gemma 2 provides a concrete example. Google trained its 2B and 9B models using a larger model's token probabilities. Its report also includes a controlled comparison: a 2B model trained for 500B tokens scored 67.7 averaged across three benchmarks with distillation, versus 60.3 when trained from scratch. That comparison helps separate the benefit of the training signal from simply choosing a smaller model. Gemma 2 report, Sections 3.2 and 5

With access to internal activations, feature distillation can also train the student to match intermediate representations. Think of comparing the working notes at selected stages, rather than only the final answer. The analogy has a limit: these are numerical activations, not readable explanations.

This sketch requires access to both models' internals, a mapping between layers, and compatible dimensions or projections. It illustrates the matching objective; it is not a complete training loop.

When only generated text is available, sequence-level distillation trains on the teacher's complete outputs. Those outputs can be translations, answers, or worked solutions. Reasoning traces are one application, not a requirement of the method. Kim and Rush, Sequence-Level Knowledge Distillation

The DeepSeek students used fine-tuning on generated training examples. The released family includes Qwen-based 1.5B, 7B, 14B, and 32B models, plus Llama-based 8B and 70B models. Their full names follow DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Llama-70B, respectively. DeepSeek-R1 distillation method

The 32B student's gap also changed by task: on MATH-500, it scored 94.3% against the teacher's 97.3%. These and the opening AIME figures are the authors' reported pass@1 results, estimating success for a single sampled answer. Their settings used temperature 0.6, top-p 0.95, and a maximum generation length of 32,768 tokens. Keep those conditions with the scores when comparing another run. Evaluation settings and results

Learning from worked solutions can improve problem solving. It doesn't establish that the student reproduces the teacher's internal reasoning. We can observe generated steps and test answers; a plausible explanation alone doesn't tell us how the model reached them.

Synthetic data also helps outside a strict teacher/student comparison. Microsoft's Phi-3-mini has 3.8B parameters and combines filtered web data with synthetic data. The report describes performance competitive with GPT-3.5 on its evaluations. It supports the value of a carefully chosen training signal, but the recipe includes more than distillation, so the result cannot isolate distillation's contribution. Phi-3 technical report

What happens when the lesson changes?

Return to the imaginary sentiment classifier. Suppose its training examples mostly contain straightforward English praise and complaints. Now customers write sarcastic reviews, mix languages, or describe a dangerous product with otherwise positive language. A good overall accuracy score can hide exactly the errors the business most needs to catch.

The teacher may handle those cases, but that ability reaches the student only if its training and capacity support it. Existing pretraining matters too: an English distillation dataset does not automatically erase a student's Japanese ability, and it does not prove that Japanese performance improved. Test the languages and situations you intend to serve.

Teacher errors can also become training examples. A fluent incorrect solution gives the student something easy to imitate and expensive to trust. For a mathematical task, I would inspect final-answer correctness and performance on unfamiliar problem variations, not award credit for a convincing sequence of steps.

There is no universal rule that a student cannot exceed its teacher. The Born Again Neural Networks experiments found students outperforming their teachers, although those students used the same architecture rather than being smaller. That is enough to reject a hard ceiling, not enough to promise that shrinking any model will improve it. Furlanello and colleagues, 2018

Turning a smaller model into a useful deployment

The memory arithmetic is easier than the quality arithmetic. At two bytes per FP16 parameter, 70 billion parameters require about 140 GB for weights; 7 billion require about 14 GB. Those are decimal gigabytes and exclude working memory and the cache used while processing a request.

The following comparison is a planning aid for hypothetical dense models, not a measured deployment:

Ten times fewer parameters gives a clear reduction in weight storage. It does not establish ten times lower latency or a fixed percentage of retained quality. A student that produces longer answers, needs retries, or sends many requests to the teacher may save less than its size suggests.

Routing can still make the tradeoff worthwhile:

The difficult part of this short example is estimated_complexity. It needs to identify the requests the student is likely to mishandle. A model's confident tone is a poor substitute for a tested routing rule. Measure the combined system, including routing errors, retries, and escalation costs.

For the sentiment application, I would set an acceptance bar before choosing a student: preserve the required recall for consequential complaints on held-out customer text, including sarcasm and supported languages, while meeting the service's latency and cost limits. Unsupported languages, ambiguous cases, and categories that fail that bar should go to the larger model or human review. Check the fallback too; a bigger model can make the same mistake.

Distillation gives you a practical way to reuse expensive training capability in a smaller model. Soft targets transfer relationships among answers; generated sequences supply examples of useful behavior. Whether that becomes a good deployment depends on the particular work that survives. Choose the smallest student that passes the task's acceptance tests, then keep measuring the cases that require escalation. That is how the smaller model earns its place.