On-policy distillation (OPD) is a post-training method in which a student LLM generates its own responses, and a stronger teacher scores them token-by-token, with the student trained to minimize the reverse KL divergence between the two models’ next-token distributions.
On-Policy Distillation (OPD) has become a popular algorithm for post-training LLMs, and almost all recent open-weight LLMs (such as Qwen3, GLM-5.3, and Nemotron-Cascade 2) have used it to achieve amazing performance.
As an example, the post-training results for Qwen3-8B showed that OPD achieved significantly better performance than RL while requiring only about 1/10th of the GPU hours.
Many engineers still confuse OPD with RL, knowledge distillation, and SFT, but you do not have to. Here is a lesson that will help you understand it well.
What is On-Policy Distillation?
LLM training takes place in three stages:
Pretraining: Teaches an LLM the basics of language (grammar, syntax, and semantic structure) and gives it foundational knowledge of the world
Mid-training: Teaches an LLM domain-specific knowledge, improves reasoning, and extends context length
Post-training: Further improves reasoning and instruction following, and aligns an LLM with human values to be helpful and not harmful
Post-training takes place in two ways:
Off-policy: Where an LLM in training (called Policy) learns from data generated from an external source, either:
Trajectories/ responses from a stronger teacher model (Knowledge distillation)
A dataset of task-specific labeled examples (using supervised fine-tuning)
On-policy: Where the policy LLM learns from its own trajectories. This can be done using either reinforcement learning (which requires a reward signal) or distillation (which requires a stronger teacher model), i.e., On-policy distillation (OPD).
OPD involves improving the capabilities of a weaker “student” model using a stronger “teacher” model.
It is called “On-policy” because the student model generates its own trajectories, which are then scored token-by-token by a stronger teacher model. This differs from conventional Knowledge distillation (KD), in which a student model learns from trajectories generated by a teacher model (an “Off-policy” approach).
KD is like a guitar teacher showing how to play like Jimi Hendrix, and a student trying to copy it. OPD, on the other hand, is like a student playing an easier song while the guitar teacher evaluates and corrects the student's playing.
With KD, a student tries to learn techniques they rarely encounter in their own playing, which can make them overconfident in their skills. With OPD, the student gradually learns from their own mistakes and improves.
Understanding OPD with an example
The process starts with two models: a stronger teacher model and a weaker student model.
The goal of OPD is to improve the student model's capabilities using its own trajectories (also called responses/ rollouts), supervised by the teacher model.
The process takes place as follows:
Pass a prompt from the prompts dataset to the student model and collect its response at each generated token position.
Pass the prompt and the student-generated response to the student and the teacher, and collect their per-token next-token probability distributions along the same trajectory.
Calculate the per-token reverse KL divergence between the student’s and teacher’s next-token probability distributions. We will discuss later why reverse KL is used instead of forward KL in this step.
Use backpropagation to update only the student’s parameters, bringing it closer to the teacher's capabilities.
Repeat these steps using the updated student model.
Let’s learn this better using an example.
Consider the following prompt from a prompt database:
“Why is space black?”
When this prompt is passed to the student model, it generates a response as follows:
“Space looks black because there is no atmosphere to scatter sunlight.”
Each token in this response is chosen from a next-token probability distribution at each step of generation.
(Also, please note that I am considering a word as a token for simplicity here. In reality, a token is typically a sub-word.)
Next, let’s look at this intermediate step in response generation with the following context:
“Space looks black because there is no”
The student and teacher models’ probabilities for the next tokens at this step are as follows (only the top three token probabilities are shown explicitly for simplicity).
The reverse KL divergence measures how these probability distributions differ and is calculated using the following formula:
where:
PS is the student model’s next-token probability distribution
PT is the teacher model’s next-token probability distribution
‘t’ is a token in the vocabulary
PS(t) is the probability the student assigns to token ‘t’
PT(t) is the probability the teacher assigns to token ‘t’
The reverse KL divergence at this token generation step is calculated as:
Similarly, the reverse KL divergence is computed between the two models’ probability distributions at each generation step and averaged to obtain the OPD loss over the student-generated response.
This loss (OPD loss) is used to update the student model’s parameters with backpropagation.
This process is repeated with the updated student model till we reach the loss minima and achieve the desired results.
Why is reverse KL divergence used as the OPD loss?
KL divergence measures how a probability distribution differs from the true probability distribution.
In our case, it measures how much the student’s probability distribution (the approximating distribution) differs from the teacher’s (the true or target probability distribution).
If the KL divergence is 0, the two distributions are identical.
The larger the value of the KL divergence, the more the two distributions differ.
When we use forward KL, i.e., DKL(Teacher || Student), as the post-training loss and iteratively minimize it, the student model tries to cover the teacher's entire output distribution. This is why forward KL is mass-covering or mean-seeking.
In simple words, when minimizing the forward KL, the teacher shows all the different kinds of answers it can generate, and the student is expected to cover/learn all of them.
With reverse KL, i.e., DKL(Student || Teacher), as the post-training loss, the student model focuses on covering the high-probability regions of the teacher’s distribution. This is why reverse KL is mode-seeking.
In simple words, when minimizing the reverse KL, the student avoids producing answers that the teacher would consider unlikely.
Reverse KL suits OPD because the student generates outputs that the teacher evaluates. And since it is mode-seeking, it helps the student focus on the high-probability behavior of the teacher model rather than trying to cover every possible behavior of the teacher.
This newsletter edition is completely free to read. Show your love by liking, restacking, and sharing it with others! ❤️
Join the paid tier today to get access to all posts in this newsletter, including:
and so many more!