PPO vs GRPO, Simply Explained

An easy lesson on two important LLM post-training algorithms.

Sep 17, 2026

LLM training is commonly done in three stages, and each stage helps the model to become more useful. These stages are as follows:

Pretraining: Teaches an LLM the basics of language and gives it foundational knowledge of the world.
Mid-training: Improves the capabilities of an LLM in intended domains (coding, health, law, etc.).
Post-training: Gives an LLM reasoning capabilities and helps align it to be a useful, human-value-aligned assistant.

Two popular reinforcement learning algorithms used in LLM post-training are PPO and GRPO. Let’s learn how these work and differ.

PPO

Proximal Policy Optimization (PPO) is an algorithm used to align LLMs to be helpful, honest, and harmless assistants. This is done using a process called Reinforcement Learning from Human Feedback (RLHF).

PPO was developed at OpenAI and later used to align a model called InstructGPT. RLHF with PPO was subsequently used to align other ChatGPT models.

PPO belongs to the family of policy-based RL algorithms known as Policy gradient. Here is how Policy gradient algorithms work:

An LLM in training generates a response for a given prompt.
A reward model assigns a reward to the response.
The reward is compared to a baseline reward the LLM is expected to get for the prompt. The difference is called the Advantage, and it tells how much better or worse the response was than expected.
The LLM’s parameters are updated to increase the likelihood of responses with a positive advantage, and to decrease the likelihood of responses with a negative advantage.

Standard policy gradient algorithms are good but not perfect and can lead to unstable alignment training.

Consider that an LLM is being trained to produce less verbose answers. Let’s say that while training, a very short response gets an unusually high reward. This will result in a large advantage and update the LLM's parameters significantly in one step.

This might make an LLM start giving one-word answers to every future prompt. And in the worst case, it might forget what it had learned in the past.

And this problem is what leads to PPO.

PPO carefully updates the LLM parameters so that the model’s behavior does not deviate too far from that of its previous version. (Hence, the term ‘Proximal’ in its name.)

An LLM’s behavior is determined by the probabilities it assigns to the tokens it generates. PPO limits how much these token probabilities can change in a single step (clipped updates), regardless of how large the advantage is. This prevents large jumps in LLM behavior in a single step and makes training more stable.

PPO uses a separate model, called the Value model, to estimate the advantage. The Value model’s job is to predict the expected reward, or “Value”, at every token in the response.

The final reward is then combined with the Value model's per-token estimates to compute an advantage for each token. This helps figure out which tokens in the response are responsible for the good or bad score.

The Value model is also trained alongside the LLM, so its predictions improve over time.

Why isn’t the reward used directly to update the parameters, and why do we need advantage?

Let’s say an LLM produces three responses to a given prompt and gets responses get rewards of 4, 7, and 10.

If the rewards are used directly, since all three are positive, the LLM parameters will be pushed towards generating all of them. This leads to noisy and slow training.

However, if we have an expected reward of 7 (coming from the Value model) for the prompt, the advantages are -3, 0, and +3. This trains the model to produce responses with higher advantage and to steer away from those with lower advantage.

GRPO

GRPO (Group Relative Policy Optimization) was introduced by DeepSeek and used to train the DeepSeekMath model to improve its mathematical reasoning. GRPO and its variants have become the standard for post-training LLMs for reasoning today.

GRPO is itself a variant of PPO that removes the Value model for advantage calculation (but keeps PPO’s clipped updates), which makes it highly memory-efficient.

Here is how it works:

For a given prompt, an LLM in training generates a group of responses.
A reward model or a verifier (when training for math and coding tasks) assigns a reward to each response.
The average and standard deviation of all rewards are calculated for the group of responses.
Advantage is calculated for each response as follows, where r(i) is the i-th response. Each token in the response gets this same advantage, which is the main trade-off of dropping the Value model.
\(A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}\)
The LLM’s parameters are updated (using a clipped update) to increase the probability of generating high-advantage responses and reduce the probability of generating low-advantage responses.

To simplify these algorithms, I have not shown:

how Generalized Advantage Estimation (GAE) is used to calculate per-token advantages in PPO, and how the Value model is trained
A reference model, which is a frozen copy of the LLM before post-training, that is used to compute a KL penalty that keeps the training LLM (Policy model) from deviating too far from it
How KL calculations are applied differently in PPO and GRPO
The full clipped objective for updating the LLM in training

A more detailed comparison diagram is shown below from the research paper describing the DeepSeekMath model. If you’re interested in learning about these algorithms in detail, please refer to the ‘Further Reading’ section.

Discussion about this post

Ready for more?