Why Does an Unbiased KL Estimate Not Give the Right Gradient? From Common KL Estimators to the Trainer-Generator Constraint in Asynchronous RL
We wrote a reading note on "Approximating KL Divergence" earlier. It discussed how large the bias and the variance of the three estimators , and are when they estimate the value of the KL divergence. This article continues that note. In training, the value of the KL is mostly only logged, or used to adjust the KL weight . What actually affects the model update is the gradient that the KL produces in the backward pass, after we write it into the loss or into the reward. An estimator that estimates the value of the KL accurately does not necessarily have a gradient that points in the direction we want. This article starts from this problem.
The article tries to answer two questions. The first question is what gradient each common form of the KL produces after the backward pass, and which direction of the KL that gradient corresponds to. The second question comes from asynchronous RL. In asynchronous RL, the generator that produces the data lags behind the trainer by some steps. In the article that adds a KL to Score Function Centering, our approach was to add a KL that pulls the trainer toward the generator. This KL can be written in several forms, and we want to understand what each form does and what it costs. The answer to the first question becomes the tool that we use to answer the second. Finally, the KL and its coefficients are computed in finite precision, so we also discuss what needs care in the computation.
This article does not give a single answer to the second question. Different forms protect against failures in different directions, and which failure is the bigger concern may differ from one setting to another. We explain what each form does, which kind of failure it protects against and what it costs. The choice is left to the reader, who can judge it in their own setting.
Before you read
To read this article, you need to know what the loss of PPO or GRPO looks like. The recap in the next section collects the three KL estimators, which are used from Section 1 on. The background on asynchronous RL and Score Function Centering is not needed until Section 4, and we recap it there.
This article may help if you have been in one of the following situations.
- You used in the GRPO loss, but you are not sure which direction of the KL it actually optimizes.
- You want to understand the difference between putting the KL in the reward and putting it in the loss.
- You work on asynchronous RL, you use a KL between the trainer and the generator, and you want to know where the different forms pull the model.
- When you logged or used the KL, you got numbers that made no sense, for example a negative value for a KL that should be non-negative.
Section 1 first explains why calling backward directly on an estimate of the KL does not, in general, give the gradient of the KL. It then uses an example with a vocabulary of only two tokens to compute how large a difference the missing term makes. To avoid such term-by-term computation every time, Section 2 writes the gradient of every KL form as a coefficient times the score. From the coefficient, we can read which direction of the KL , and correspond to when they are used as a loss. The same coefficient also has a different effect in the reward than in the loss, and Section 3 discusses this difference. Sections 1 to 3 all assume that the samples come from the current model. Section 4 removes this assumption. It first recaps the background of asynchronous RL and Score Function Centering, and then discusses two cases separately, GRPO with several updates on one batch and asynchronous RL. In asynchronous RL, there is more than one form that brings the trainer and the generator closer. Section 5 writes all of them as modifications of the advantage. Section 6 then compares how they constrain tokens whose probability is falling and tokens whose probability is rising, where they finally pull the distribution, and what each of them costs. Finally, Section 7 discusses how to compute the KL and its coefficients in finite precision, and Section 8 concludes.
All figures in this article are interactive. Move the mouse over a figure to read the values, and click a legend entry to hide a curve. In a figure with a slider, drag the slider to change a parameter.
Recap
So that this article can be read on its own, we first recap the three estimators that are used later.
Suppose we want to estimate , but we only have samples drawn from . Here is the distribution we sample from, and is the other distribution. Write . In his blog post "Approximating KL Divergence", John Schulman gives three estimators:
is unbiased, but its variance is large. has a small variance, but at the price of a bias. adds to the control variate , whose expectation is 0, so it stays unbiased and also has a small variance. These three estimators can be used for any two distributions. Whenever we use them later, we say which distribution is in the place of and which distribution is in the place of .
1. Why does differentiating the KL estimate directly not give the gradient of the KL?
TL;DR: The KL is an expectation, and appears both in the estimate that is averaged and in the distribution that is sampled. Calling backward directly on the KL estimate only differentiates the estimate itself. The part of the gradient that comes from the change of the sampling distribution with is lost, and more samples cannot recover it. So an estimator that is unbiased for the value of the KL is not guaranteed to give the correct gradient when it is used as a loss.
This section first shows which term of the gradient is lost, and then uses an example with a vocabulary of only two tokens to compute how large a difference the missing term makes. Before that, we fix the notation used in Sections 1 to 3.
To keep things simple, we fix a prefix and look only at the distribution of the next token. Write for the logits that the model outputs on this prefix. It is a vector whose length is the vocabulary size. The model being trained is , and the probability that it gives to token is . The reference distribution in the KL regularization is the reference model, written . is usually the model before RL starts, for example the model after SFT. It stays fixed throughout training, so it differs from except at the very beginning of training. For a token , write its log ratio as
Write the score as , which is the gradient of the log probability of token with respect to the model parameters. The examples below differentiate only with respect to the logits , and then the score has a simple form. Write for the one-hot vector of token , that is, the vector that equals 1 at the position of and 0 at every other position. We also arrange the probabilities of over the whole vocabulary into a vector, and still write it as . Since , its partial derivative with respect to is when , and when . Arranged into a vector, these partial derivatives are
The KL divergence has two directions:
In RLHF and GRPO, the one that we usually want to constrain is the first. This article follows the naming of our earlier article that adds a KL to Score Function Centering. A KL with in the second place, such as , is called the reverse KL. A KL with in the first place, such as , is called the forward KL. Some RLHF papers and codebases use the opposite names. The appendix at the end of the article lists how the two naming conventions correspond.
In the three formulas of the recap, is the distribution we sample from and is the other distribution. Here we want to estimate , and the samples come from . So is in the place of , and is in the place of . Then and . After this substitution, the three estimators, written as functions of , are
Sections 1 to 3 all use this notation. They use only and do not use on its own.
1.1 Which term does backward miss?
To discuss the three estimators at once, we write for the value of any one of them on the sample . For example, with , . The subscript shows that it depends on the model parameters. For unbiased estimators such as and , the KL equals , so the gradient of the KL that we want is the gradient of this expectation. Written as a sum, the expectation is
In this expression, appears in two places. One is the estimator itself. The other is the weight , which is the probability that token is sampled. Differentiating the product gives two terms. We call the term that differentiates the estimator term. We call the term that differentiates the sampling term, after we rewrite it with . This gives
A common implementation computes the KL term as follows. It first samples tokens from . It then computes for each token, averages these values into , multiplies the average by and adds it to the loss. Finally it calls backward. The gradient from backward is , and its expectation is exactly the estimator term above.
The sampling term comes from differentiating the weight . In the code, this weight does not appear in the loss as a number. The sampling applies this weight instead. A token with a larger probability is sampled more often, so it takes a larger share of the average. The sampling step, however, outputs the index of a token. When changes slightly, the sampled index either stays the same or suddenly jumps to another token. So the derivative of a single sampling result with respect to is either 0 or does not exist, and automatic differentiation does not go through this step. The effect of on how often each token is sampled shows up only in the average over many samples, and no single backward pass contains this effect. Therefore backward differentiates only the expression and treats the sampled as a constant, and this is how the sampling term is lost. Note that may also contain . For example, is . Backward does differentiate this occurrence, and it belongs to the estimator term. Only the that acts as the weight is lost.
The sampling term has the same form as a policy gradient. If we view as the reward that token receives, the sampling term is , which is the REINFORCE gradient. The problem that the policy gradient solves is how to take the gradient of an expectation when the sampling distribution depends on . Readers who are not familiar with this derivation can see Section 1.1 of the article that introduces Score Function Centering. So differentiating the KL estimate directly computes only the gradient of the estimator itself, and drops the whole gradient that comes from the change of the sampling distribution.
Because the lost part is a whole expectation term, more samples only make the estimate of the estimator term more accurate. They cannot bring back the sampling term, so this bias does not depend on the number of samples. Tang and Munos (2025) and Zhang et al. (2025) both discuss this problem in detail, and they point out that many open-source implementations are affected by it.
1.2 How large is the difference when this term is missing?
Below, we compute three gradients exactly in an example with a vocabulary of only two tokens, to see how large a difference the missing sampling term makes.
Suppose that the vocabulary has only two tokens, A and B. The current model has , and the reference distribution has . To keep the formulas short, write and . We compare three quantities. All of them are derivatives with respect to , the logit of A, and all of them take the exact expectation over the sampling.
The first quantity is the derivative of the forward KL itself with respect to , which is the result we want. With only two tokens, . From above, the derivative of with respect to is , and the derivative of with respect to is . So the derivatives of and with respect to are and , respectively. The derivative of or itself equals the probability times the derivative of its log probability, which gives and . When we differentiate the two products, the two terms that differentiate are , and they cancel exactly. These two terms are the estimator term of Section 1.1. What remains are the two terms that differentiate and , and they form the sampling term:
The second quantity is the expectation of the gradient that backward gives when is used as the loss. Since , its derivative with respect to is just . From , this derivative is when A is sampled, and when B is sampled. The expectation under is .
The third quantity is the expectation of the gradient that backward gives when is used as the loss. Since , its derivative with respect to is , where . When A is sampled, the derivative is . When B is sampled, the derivative is . The expectation under is .
As estimates of the value of the KL, and are both unbiased. But when they are used directly as the loss, their expected gradients are 0 and 0.2, and neither of them equals the 0.178 of the forward KL. All three numbers are exact expectations, so the difference exists no matter how large the batch is. is the most extreme case. Its gradient averages to 0, which means that this KL term does not pull the model toward the reference distribution at all.
This difference is not caused by the particular value 0.7. Figure 1 lets take every value between 0 and 1 and plots the same quantities. The vertical line in the figure marks the example above. Drag the slider to change . The figure also shows the result of as the loss and the gradient of the reverse KL. Section 2.2 explains why the curve of coincides with the curve of the forward KL, and why the curve of the reverse KL coincides with the curve of .
The 0.2 from equals exactly . The next section gives a single method that shows directly which gradient each form corresponds to, and Section 2.2 uses it to explain where the 0.2 comes from.
2. What gradient do , and each give when used as a loss?
TL;DR: Once the gradient of each KL form on a single sample is written as a coefficient times the score, comparing the coefficients tells us which KL the form optimizes. When the samples come from the current model, as a loss has an expected gradient of 0. gives the gradient of , which this article calls the forward KL. gives the gradient of , which is the reverse KL. So InstructGPT-style PPO, which puts in the reward, optimizes the forward KL. GRPO puts in the loss, and under the approximation that the samples come from the current model, it optimizes the reverse KL.
The three numbers in Section 1 were computed one at a time. This section gives a single method that writes the gradient of every KL form on a single sample as a coefficient times the score. Section 2.1 introduces the method. With it, Section 2.2 reads off the gradient that each of , and gives as a loss, and Section 2.3 compares the KL in PPO with the KL in GRPO.
2.1 How can we read the gradient of an estimator?
, and depend only on , so all of them can be written as . does not change with , so . For any form , the gradient that backward gives on a single sample is
This gradient is a scalar coefficient times the score. In gradient descent on the loss, a positive coefficient lowers the log probability of this sample, and a negative coefficient raises it. The absolute value of the coefficient sets the size of the change. Comparing different forms means comparing their coefficients.
When the samples come from the current model, a gradient with the coefficient has the expectation . The score has expectation 0, so adding to any constant that does not depend on does not change this expectation. This is also why a baseline can be subtracted in the policy gradient. We discussed this in detail in Section 2.1 of the article that introduces Score Function Centering, and we use it repeatedly later.
Next, consider what the gradient of the forward KL should be. Take in the formula of Section 1.1. The estimator term is , so only the sampling term remains in the gradient of the forward KL. In other words, when the samples come from the current model, a coefficient equal to , or to plus any constant, gives the exact gradient of the forward KL. The next section uses this criterion to check the three estimators.
Here "exact" refers to the expectation. Adding a constant to the coefficient, or subtracting one, does not change the expectation, but it changes the variance of the gradient. As with subtracting a baseline in the policy gradient, we can subtract from the coefficient a quantity that does not depend on the token just sampled, to lower the variance of the gradient. The of Section 5.1 also uses this idea. Note that the variance discussed in the earlier note is a different quantity. It is how much the KL estimate varies from sample to sample, which is not the variance of the gradient discussed here.
2.2 What are the coefficients of , and ?
Start with . Since , the coefficient is , and the expected gradient is . As an estimate of the value of the KL, is unbiased, but as a loss, the expectation of its gradient is always 0. On a single batch its gradient is not 0, but on average these gradients point in no direction and only add noise to training. This is why gives 0 in Section 1.2.
Next comes . Since , the coefficient is . By the criterion of Section 2.1, it gives the exact gradient of the forward KL. In the example of Section 1.2, when A is sampled, and when B is sampled. Multiplying each by the derivative of with respect to and taking the expectation under gives exactly 0.178. This is also why the dashed line of in Figure 1 coincides exactly with the curve of the forward KL.
This is the opposite of how behaves as an estimate of the value. As an estimate of the value of the KL, is biased. Schulman's blog post shows that when the two distributions are close, its expectation agrees with the KL only up to second order. As a loss, however, gives an exact gradient when the samples come from the current model. The reason is that the estimator term has expectation 0, so only the sampling term remains in the full gradient of the forward KL, and the backward pass of gives exactly on a single sample. In other words, the backward pass of happens to compute the term that misses.
Finally, consider . Since , the coefficient is
Its expected gradient is
The gradient of is also . So when is used as a loss, its expected gradient is the gradient of the reverse KL. In the example of Section 1.2, the gradient of the reverse KL with respect to the logits is , and its component on A is . This is the number that gives. In Figure 1, the dashed line of coincides exactly with the curve of the reverse KL.
When is close to 0, , which agrees with the coefficient of to first order. For this reason, Liu et al. (2025) describe the loss as a first-order approximation of . This does not contradict the statement that gives the gradient of the reverse KL. When the model is not far from the reference distribution, the gradients of the two directions of the KL are close anyway, and the farther apart the two distributions are, the larger the difference between the two gradients. Figure 1 shows this too. The two curves almost coincide near , and they separate more as moves away from this point.
The coefficient of is not symmetric on the two sides of . Figure 2 plots the coefficients of the three estimators on a single sample. On the side , where the model gives a token a higher probability than the reference distribution does, the coefficient of is at most 1. On the side , where the model gives a token a lower probability than the reference distribution does, the absolute value of the coefficient grows exponentially, like . For example, at the coefficient of is 2 and the coefficient of is 0.865. At the coefficient of is and the coefficient of is .
This is the coefficient on a single sample. A token with far below 0 is rarely sampled under . After we multiply by the probability that it is sampled, the expected gradient on its logit is . Gradient descent pulls the token back, but the size of this pull is never more than . So the exponentially growing coefficient does not bring a large pull in expectation. What it mainly brings is variance. When such a token happens to be sampled, it produces a very large gradient. Section 4.1 shows that this is the criticism of in the DeepSeek-V3.2 technical report. Section 6 analyzes how large the pull is on each side, in the setting with a generator.
2.3 How does the KL in PPO differ from the KL in GRPO?
Besides serving as a loss, the KL can also be put in the reward. InstructGPT-style PPO (Ouyang et al., 2022) does this. It multiplies the of each token by and subtracts the product from the reward of this token. Here is treated as part of the reward and is not differentiated. It affects the gradient through the policy gradient, as a coefficient that multiplies the score. The coefficient of the KL part is , so in expectation it is the same as used as a loss, and it also gives the gradient of the forward KL. This result holds only at a single position. Section 3 discusses how the two differ over a whole sequence.
With these results, we can look at the KL in PPO and GRPO again. In the earlier note we wrote that " is exactly the piece where, in practice, GRPO differs from PPO in how KL is estimated (PPO uses )". We can now state this more completely. InstructGPT-style PPO puts in the reward, and its gradient corresponds to the forward KL. GRPO (Shao et al., 2024) puts in the loss, and under the approximation that the samples come from the current model, its gradient corresponds to the reverse KL. Besides using different estimators, the two methods also optimize different directions of the KL. As Section 2.2 said, when the model is not far from the reference model, the gradients of the two directions are close, so this difference may not matter much in practice. The DeepSeek-V3.2 technical report later corrected this term of GRPO, and Section 4.1 describes the correction.
So far, we have discussed only the gradient at a single position. in the reward and in the loss have the same expectation at a single position, but in real training the two placements still differ. The next section discusses this difference.
3. What is the difference between putting the KL in the reward and putting it in the loss?
TL;DR: When the KL is in the reward, it goes through the normalization of the advantage and the PPO clip together with the task reward, so the same has a different actual effect on different prompts. The KL is also passed to earlier tokens through the advantage. This gives the exact gradient of the sequence-level KL, but with a larger variance. When the KL is in the loss, is the actual weight of this term. Each token constrains only the distribution at its own position. The variance is smaller, but the result is not the gradient of the sequence-level KL.
Putting the KL in the reward means subtracting , usually its detached value, from the reward of the -th token, and then computing the advantage together with the task reward. The classic implementation of PPO does this. Putting the KL in the loss means computing the advantage from the task reward only, and adding the KL to the loss as a separate term that is differentiated, for example . GRPO uses this second approach.
Section 2.3 showed that at a single position, in the reward and in the loss have the same expected gradient. But the two placements still differ in two ways in real training. The first difference is that, in the reward, the KL goes through all the processing of the advantage together with the task reward. The second difference is that, in the reward, the KL is passed along the sequence to earlier tokens. The next two sections discuss them in turn. Both differences matter again in the second half of the article. Section 5 shows that under Score Function Centering, every KL that brings the trainer and the generator closer can be written as a modification of the advantage, and the same two issues need attention there.
3.1 How is the KL rescaled once it is in the reward?
Once the KL is in the reward, the advantage carries two kinds of information at the same time, namely how well the task was done and how close the response is to the reference distribution. A response may get a higher advantage only because it is closer to the reference distribution. Besides this, every processing step that comes after the advantage also acts on the KL. For example, suppose that the KL is added to the reward, and that the advantage is then normalized by the standard deviation of the group, which is the normalization that GRPO uses. Then the KL on each prompt is divided by the standard deviation within the group of that prompt. A prompt whose rewards differ little within the group has a small standard deviation, so its KL is enlarged. A prompt whose rewards differ a lot within the group has its KL reduced. So the same has a different actual effect on different prompts. The PPO clip also acts on the KL. If the gradient of a token is clipped away, the gradient of its KL part disappears with it.
When the KL is in the loss, none of this processing affects the KL, and is the actual weight of the KL term.
3.2 Why is the KL passed to earlier tokens once it is in the reward?
First, recall how the advantage is computed from the reward of each token. Write for the reward on the -th token, and for the value that the critic estimates for the state at step . GAE (Generalized Advantage Estimation) first computes the TD residual of each step, . It then sums the residuals of this step and of all later steps, weighted by powers of :
Here is the discount factor, and controls how far ahead the advantage looks. With it looks only one step ahead, and the closer is to 1, the farther it looks. We introduced GAE in detail in Section 8.2 of the article that introduces GRPO. With , the terms of neighboring steps cancel in pairs, and . This is the sum of all rewards from the -th token to the end of the sequence, minus a baseline.
When the KL is in the reward, the reward of the -th token gets an extra term . By the formula above, the advantage of the -th token then gets an extra , which contains the KL of every token from the -th token to the end of the sequence. In the terms of Section 2.1, the coefficient of the KL part on the -th token is
This coefficient corresponds to the gradient of the sequence-level forward KL. The reason is the same as for the sampling term in Section 1.1. The choice of the -th token changes the prefixes of all later tokens, and the probabilities of these prefixes also depend on . So an early token is responsible for all the KL that is produced after it.
When the KL is in the loss, the coefficient of each token depends only on its own . For example, the coefficient of is , and the coefficient of is . This constrains the distribution of the next token separately on each fixed prefix, but it ignores the effect of the current token on later prefixes. Tang and Munos (2025) call the gradient obtained in this way a partial gradient.
So in the reward and a per-token in the loss are equivalent only at a single position. Over a whole sequence, the coefficient of in the reward is the sum of over the current token and all later tokens. It gives the exact gradient of the sequence-level KL, but the longer the sequence, the more this sum fluctuates, and the larger the variance of the gradient. The per-token in the loss uses only the of each token. Its variance is small, but it is not the gradient of the sequence-level KL.
To get the gradient of the sequence-level KL in the loss, we can apply to the sum of the log ratios over the whole sequence, that is, . Its gradient is , so the score of the -th token is multiplied by the of all positions. The with are already fixed before the -th token is sampled, so their products with the score of this token have expectation 0. In expectation, this loss is therefore the same as in the reward, but its variance is even larger.
Sections 1 to 3 all assume that the samples come from the current model . This does not hold when GRPO updates several times on one batch, and it does not hold in asynchronous RL either. The next section discusses what happens when the samples do not come from the current model.
4. What happens when the samples do not come from the current model?
TL;DR: This section removes the assumption that the samples come from the current model. When GRPO updates several times on one batch, the samples come from , and the gradient of as a loss is neither the gradient of the forward KL nor the gradient of the reverse KL. DeepSeek-V3.2 multiplies by the ratio and lets this ratio be differentiated, and only then is the result the exact gradient of the forward KL. In asynchronous RL, the samples come from a fixed generator. The sampling distribution does not depend on , so no term is lost, and calling backward directly on or gives the exact gradient of .
In Sections 1 to 3, the KL always compares the model with the reference model , and the samples always come from the current model . From this section on, the samples no longer come from . Before we continue, we put the four distributions of this article side by side.
- is the model being trained. It is updated at every step.
- is the reference model, usually the model before RL starts. It stays fixed throughout training. The KL regularization of PPO and GRPO compares the model with it, to limit how far the model moves away from the initial model.
- is the model that generated the current batch of data in PPO and GRPO, that is, the generator of this batch.
- is the model that generates the data in asynchronous RL. It has the same role as , but it lags further behind.
How these distributions relate to each other depends on how the training is run. In fully on-policy synchronous training, each batch is used for only one update, and the at the update is the model that has just generated this batch. So and have the same parameters, and the only difference is that is not differentiated. is different. It stays fixed from the start of training while keeps being updated, so even in synchronous training the two are the same only at the very beginning. When GRPO updates several times on one batch, from the second update on, has already taken a few update steps since generated the batch, and the two are no longer the same. In asynchronous RL, the generator produces data while the trainer is being updated, and lags behind by a few steps or even hundreds of steps.
Score Function Centering deals with this lag in asynchronous RL. From Section 4.2 on, the rest of the article works in this setting, so we first recap how Score Function Centering works, and the KL that we added on top of it. Our earlier articles wrote the trainer and the generator as and . This article writes them as and .
Score Function Centering. The score in the policy gradient has expectation 0 only when comes from , and this is what allows a baseline to be subtracted from the advantage. When comes from , this expectation is no longer 0, and the policy gradient gets an extra term. This term equals the mean advantage times the expectation of the score under . Its direction on the logits is , and its size and sign are both set by the mean advantage, which says nothing about which token is better. Score Function Centering, SFC for short below, centers the score at instead, and so removes this term. If we look only at the part that differentiates the logits at the current position, the ordinary score is , and the score after SFC is . Here is the computed in Section 1.
Computing the SFC score needs the full distribution of the generator at this position. In practice, the generator sends back only its most probable tokens and their probabilities. This part is called the head, and the rest is called the tail. There are two ways to handle the tail. One is to sample a few more tokens from the tail to estimate it. The other is to record only the total probability of the tail, and our earlier experiments used this second way by default. The detailed derivation is in our article that introduces SFC.
Adding a KL to SFC. SFC does not limit the distance between the trainer and the generator. The larger the distance, the more the update computed on the samples of the generator differs from the update that the current model should get. In the article that adds a KL to SFC, we derived an upper bound on this difference that is proportional to , so we added a on top of SFC. Its gradient with respect to the logits is , and it can be computed with a sparse estimate. For a token sampled from the generator, this estimate takes the value on the logit of itself, and 0 on every other logit. Its expectation is exactly .
In addition, the KL weight of each token decays with the age of the token. The age of a token is the number of steps by which the generator version that produced it lags behind the current trainer. In our earlier experiments we called it staleness. Concretely, the KL weight equals divided by the square root of the age of the token, and a token with age 0 gets the weight 0.
This section discusses two cases in which the samples do not come from the current model. The first case is GRPO with several updates on one batch. The samples come from , and the KL still compares the model with . The second case is asynchronous RL. The samples come from , and the KL that we added on top of SFC compares the model with itself. This KL limits how much the model is updated within each generator version. In fully on-policy synchronous training, the generator is itself, so apart from numerical error this KL is always 0. Section 7.2 discusses the problems that numerical error causes.
The analysis of the two cases uses two earlier results. One is the split into the estimator term and the sampling term in Section 1.1. The other is reading a gradient through its coefficient, as in Section 2.1. The second case is the basis of the later sections.
4.1 How should be corrected when GRPO updates several times on one batch?
When one batch is used for several updates, from the second update on, the samples come from instead of the current . Every expectation above must now be taken under , and the earlier results no longer hold exactly. Take the loss as an example. Its expected gradient becomes , which is neither the gradient of the forward KL nor the gradient of the reverse KL.
To see how this term should be corrected, we first need an identity about . For every sample,
The first term on the left is the gradient that backward gives when is used as a loss, and it corresponds to the estimator term of Section 1.1. The second term is the part that the policy gradient gives when is used as a reward, and it corresponds to the sampling term that was lost in Section 1.1. Their sum is exactly , the single-sample form of the gradient of the forward KL. The identity shows that calling backward directly on keeps only the estimator term, and that adding the sampling term back gives the gradient of the forward KL.
The DeepSeek-V3.2 technical report corrects this term in its Equation (7). It multiplies by the ratio of the current model to the old model:
If is differentiated, then
The last step uses the identity above. Taking the expectation under gives , which is the exact gradient of the forward KL.
Seen through its coefficient, this correction does two things at once when is differentiated. The first is to correct the sampling distribution, by turning an expectation under into an expectation under . This is what the ratio is normally for. The second is less obvious. The derivative of with respect to is , and its effect is to add the sampling term back. So even at the first update, where and are identical and the value of is 1, the gradient of changes from the reverse KL to the forward KL as long as is differentiated. Conversely, if is detached, the result is still the gradient of the reverse KL.
The DeepSeek-V3.2 report also explains what was wrong with the original . If a sampled token has a much lower probability under the current model than under the reference model, the gradient of gives it a large weight with no upper bound, which makes the gradient noisier. This corresponds to the side in Section 2.2, where the absolute value of the coefficient of grows exponentially.
4.2 When the samples come from a fixed generator, why is differentiating directly correct after all?
In the asynchronous RL setting that we discussed earlier, the samples come from , and the KL also compares the model with . What we want to constrain is
Unlike before, the sampling distribution does not depend on . The sampling term of Section 1.1 comes from the dependence of the sampling distribution on . This dependence is now gone, so nothing can be lost.
From this section on, the KL compares the model with , so we redefine the log ratio accordingly as , and we write . When we apply the three estimators of the recap, the samples come from . So is in the place of and is in the place of , and this is exactly the of the recap.
Now call backward directly on , that is, on . The gradient on a single sample is , and its expectation
is the exact gradient. In logit space this gradient is , and the update direction of gradient descent is . Minimizing this KL with gradient descent means maximizing on the tokens that the generator sampled, which is SFT on the samples of the generator.
also works. Since , the gradient on a single sample is . Its expectation
is the same as the expectation for . Compared with , has the extra term . Under , this term has the expectation , so it is a control variate. When and are close, it can lower the variance of the gradient. When the value of the KL is estimated, uses a control variate to lower the variance of the value. Here the same idea lowers the variance of the gradient.
Compare this section with Section 2.2. The code in the two places is almost the same. Both call backward on the difference between the log probability under and the log probability under another distribution, and they differ only by a sign. When the samples are drawn by the current model itself, the expected gradient is 0. When the samples come from a fixed generator, the same code gives the exact gradient of . The only difference between the two cases is whether the sampling distribution depends on . Also, by the naming of Section 1, is a reverse KL, like the whose gradient gives as a loss in GRPO.
This update direction also appeared when we introduced SFC. As we discussed in the article that introduces SFC, if SFC is not used in asynchronous RL, the mean advantage multiplies the same update direction , and the size and sign of this term are both set by the mean advantage. SFC removes this term from the policy gradient. Adding then adds the same direction back, but with a weight that we choose ourselves. This is also why we chose the direction before.
5. Under Score Function Centering, how does a KL that brings the trainer and the generator closer act on the advantage?
TL;DR: Write . Under Score Function Centering, adding a trainer-generator KL with the weight is equivalent to the following change. After the advantage is fully computed, subtract from the advantage of each sampled token times a coefficient that depends only on . The coefficient of is . is half the variance of under , and its coefficient is . The coefficient of is approximately . The three coefficients agree to first order near , so when the trainer and the generator are close, the forms differ little.
Section 4.2 showed that when the samples come from a fixed generator, the gradient of can be computed exactly. However, an exact gradient only means that we optimize this KL exactly. It does not mean that this KL is the most suitable constraint. There are several other forms that pull the trainer toward the generator. They act in about the same way when the trainer and the generator are close. When the two are far apart, the forces that they put on tokens whose probability is falling and on tokens whose probability is rising are very different. To compare these forms, this section works in the setting of Score Function Centering (SFC) and writes each form as subtracting times a coefficient from the advantage. Section 5.1 derives this expression, and Section 5.2 uses an example with a vocabulary of only three tokens to see how much the forms differ when the trainer and the generator are close.
5.1 How can a KL term be written as a modification of the advantage?
Under SFC, the score is the version that is centered at , and its part on the logits at the current position is . From this section on, "coefficient" means the quantity that multiplies this score. Suppose that the samples come from and that the coefficient of each sample is . Arrange the values of over the whole vocabulary into a vector . Then the expectation is
Write for the vector whose components all equal 1. Since , adding the same constant to every component of does not change the result. This is the baseline invariance under SFC.
The policy gradient term can also be written in this way. Write for the expected advantage after choosing token on the current prefix, and arrange these values into a vector . The coefficient of the policy gradient term is the advantage of the sampled token, and the expected update direction that it gives is . Suppose that the expected gradient of some KL used as a loss can also be written as . Then, after this KL is added with the weight , the expected update direction of gradient descent is . In other words, adding this KL is equivalent to changing the advantage from to .
When this is implemented as a modification of the advantage, the two issues of Section 3 need attention. The equivalence above holds at each position separately, with as the actual weight of this term. So the modification should be applied after the advantage is fully computed, that is, after the normalization within the group. It should act only on the current token, and it should not be summed onto earlier tokens along the sequence.
Below we look at the of the three forms in turn. The first form is . Section 4.2 showed that its gradient in logit space is . Also, , where means component-wise multiplication. So
So under SFC, adding with the weight is equivalent in expectation to changing the advantage of each sample from to . The sparse estimate that we recapped at the start of Section 4 also uses the number , but it multiplies , so it updates only the logit of the sampled token. The two approaches give different gradients on a single sample, but both have the expectation .
The second form starts from the wish that the coefficient should be itself. Compared with , grows only logarithmically, so it does not become very large even when is large. One can check that the coefficient of the following objective under SFC is exactly . The objective is half the variance of under :
Its gradient is
uses the mean of , while this objective uses the mean of the square after the mean of is subtracted. To place it next to the first three estimators, we write it as below. Schulman's blog post has only to , and the name is one that we chose in this article for convenience. When and are very close, agrees with up to second order, so it can also be seen as a form of the KL. In the gradient, the mean of under is subtracted from , which is the same as subtracting a baseline in Section 2.1. Under SFC, it makes no difference whether this mean is subtracted, because .
So under SFC, is equivalent to changing the advantage to . It is simple to implement and needs no separate loss term. On each sampled token, we only need to modify the advantage with the detached value of . KLPO (Zhang, 2026) discusses the same objective. It calls this approach token regression. In its Appendix D, it writes the corresponding objective as the advantage-weighted log ratio plus times the variance of the log ratio.
Note that must be used together with SFC. If the score is still the ordinary , the same code implements the version without the mean subtracted, that is, the gradient of . The coefficient of this objective under SFC is , which has an extra term compared with .
The third form is the KL in the other direction, . When the KL compares the model with the reference model, this is the direction that RLHF usually constrains, so it is worth seeing what happens when it is used with the generator. It is the forward KL that compares the model with , and by Section 2.1 its gradient is . When the samples come from , multiplying by the ratio turns this expectation into an expectation under . So the ordinary score with the detached coefficient gives the exact gradient:
Under SFC, the corresponding coefficient is . When and are close, is a small quantity of second order, and this coefficient is approximately .
To summarize, under SFC the coefficient of is , the coefficient of is , and the coefficient of is approximately . All three coefficients are functions of that equal 0 at and agree to first order near . Each form can be implemented by modifying the advantage, using only the and the of the sampled token. The exact coefficient of also needs an estimate of itself.
5.2 When the trainer and the generator are close, do these forms differ much?
The three coefficients of Section 5.1 agree to first order near . This means that when the trainer and the generator are close, the gradients that they give should differ little. Below, an example with a vocabulary of only three tokens shows this, and it also shows how the forms differ in the details.
Let and . Then , , and . The two directions of the KL give and . The expected gradients of the three forms on the logits are listed below, and Figure 3 plots them together.
- The gradient of is .
- The gradient of is .
- The gradient of is .
Under gradient descent, the logit of the first token rises and the logit of the third token falls. The three forms agree on this, and all of them pull back toward . The difference is on the second token, whose is 1. Under it does not move at all. Under it is pushed down slightly. Under it is instead raised slightly. The reason is that each form compares with a different baseline. compares with its mean under , which is , so a token with is relatively high. compares with its mean under , which is , so the same token is relatively low.
In this example the forms differ little, because the trainer and the generator are not far apart. How far apart they get depends on the size of relative to the advantage. When is clearly larger than the differences between advantages, the KL keeps near 1, and the forms differ little. When is of the same order as the differences between advantages, or smaller, moves far from 1. Then the shapes of the coefficients far from make the forms behave very differently. In our earlier experimental setup, a typical token has a of only about 0.04, while the differences between advantages can be of order 1. This is the second case, and Section 6.1 explains where these two numbers come from. The next section discusses this case.
6. How do the three forms constrain tokens whose probability falls and tokens whose probability rises?
TL;DR: The three forms differ mainly when is far from 1. For a token whose probability is being pushed down, the pull of has no upper bound as the probability falls, so can keep these tokens from being pushed to 0. The pull of has an upper bound, and the pull of goes to 0. So when is small, both of these forms may let tokens with a low advantage be pushed all the way down to 0. For a token whose probability is being raised, the situation is the reverse. gives the weakest constraint, and gives the strongest. Which form is more suitable depends on which kind of token carries more risk in training, and on the size of relative to the advantage.
Section 5 wrote the three forms as subtracting times a coefficient from the advantage, with the coefficient for , for , and approximately for . When the trainer and the generator are far apart, the shapes of these three coefficients determine how the forms behave. We look at two kinds of tokens. The first kind consists of tokens whose probability is being pushed down, that is, tokens with . The second kind consists of tokens whose probability is being raised, that is, tokens with . Section 6.1 first compares the pull of the three forms on a single token, and Section 6.2 then uses a simplified model to see where the forms finally pull the distribution. Section 6.3 discusses the cost of each form, and Section 6.4 discusses what to consider when choosing a form.
6.1 When the probability of a token keeps being pushed down, can the three forms pull it back?
The weight of the KL is , but the actual effect of the KL on training also depends on the coefficient that multiplies, and this effect has to be compared with the size of the advantage. For this, we look at the "force" on the logit of a token under SFC, where the force means the expected update on this logit. Write , where is the expected advantage after choosing token , as defined in Section 5.1. By the result of Section 5.1, the expected update of the policy gradient term on is . This quantity depends only on , and not on the current value of . In other words, while the generator is fixed, SFC keeps pushing or pulling this token with the same force. This differs from the case in which the samples come from the current model. In that case, the expected update of the policy gradient on is , so the lower the probability of the token, the smaller the push.
The forces with which the three forms pull this token back are as follows.
- The pull of is . As , this force approaches and does not grow any further. This is the same as for used as a loss in Section 2.2, where the pull also never exceeds .
- The pull of is . As , keeps growing, so this force grows linearly with how far the logit has fallen, and it has no upper bound.
- The pull of is . When it is written as times a coefficient, the coefficient is approximately . The function reaches its minimum, , at , and it returns to 0 as decreases further. So this form pulls the token back with a force of at most about , and the lower the token is pushed, the weaker this force.
Now compare the push with the pull. For , the force that pulls back a token that is being pushed down is never more than , while the force that pushes it down is . As long as , the push stays larger than the pull. This KL cannot stop the probability of this token from falling further, and the token is finally pushed toward 0. When the samples come from the current model, the push becomes smaller together with , and a bounded pull is enough to hold the token. Under SFC the push does not become smaller, so a bounded pull may not be enough to hold the token.
Figure 4 plots the pull of the three forms as a function of , in units of . It leaves out the two small quantities and . The lower a token is pushed, the farther left it moves in the figure. The dashed line is the push , which you can change with the slider. Where a curve crosses the dashed line, the pull balances the push and the token stops. If a curve stays below the dashed line, that form cannot hold the token.
The case can occur in real training. In our earlier experimental setup, the KL weight equals divided by the square root of the age. On the 4-prompt slice of Countdown, that is, when each update of the trainer uses the rollouts of 4 prompts, the mean age of a token is about 50, and a token with age 50 has . When those experiments computed the advantage, a correct response scored and a wrong response scored , and then the mean within the group was subtracted. So the advantage of a single response is of order . The of a single token is usually smaller than that, because for most tokens, switching to a different choice does not change the outcome of the response. But on the few tokens that determine the outcome of the response, can come close to this order, far above 0.042.
However, in those experiments, SFC plus never collapsed, and this does not contradict the analysis above. The comparison above assumes that the push lasts. In real training, each slice of rollouts is used for only one update, the generator keeps being refreshed, and the same prefix is rarely updated again and again, so the push does not last. Also, a token that is pushed down to 0 does not necessarily make training collapse. The upper bound of the pull shows that once the push persists, the protection that this KL can provide is limited.
Textbooks often say that is zero-avoiding. Whenever and , the value of the KL goes to infinity, so minimizing it avoids pushing any part of to 0. This statement is true for the value of the KL. But the value of grows only like , which is roughly linear in the logit, and its gradient with respect to the logit is bounded. Against a push that stays constant, a value that goes to infinity does not guarantee that the token is kept away from 0.
6.2 If training continues, where do the three forms pull the distribution?
Section 6.1 looked only at the force on a single token. To see where the three forms finally pull the whole distribution, we use a simplified model. We fix a prefix and a generator version, and we assume that the expected advantage of each token stays fixed. Then we let the expected update continue indefinitely and see where it stops.
In optimization, a point where the gradient equals 0 is usually called a stationary point. The update direction here is not necessarily the gradient of some objective function, so by a stationary point we mean a point where the expected update equals 0, which is a fixed point in the sense of dynamical systems. Once a stationary point is reached, the expected update no longer changes , and if the updates converge, they stop there. A stationary point may not be reachable. The probability of some tokens may keep falling during the updates and never stop, and then what we care about is the limit. The analysis and the examples below are all about stationary points.
Under SFC, the update direction is , where is the coefficient of the form. When every component of is greater than 0, the only vectors with are constant vectors. So the condition for a stationary point is that is the same for all tokens.
For , , and the stationary point is
The stationary point is a linear function of the advantage. It is also the stationary point of , where . In other words, within one generator version, SFC plus has the same effect as a regularization. regularization has also been discussed for LLM alignment. Huang et al. (2024) argue that it suppresses the overoptimization of an imperfect reward better than KL regularization does. This stationary point is a valid distribution only when for every token. Otherwise, the tokens with keep being pushed toward 0, which agrees with the result of Section 6.1.
For , , and the stationary point is
This is the classic optimal solution of , and it has the same form as in RLHF and DPO, where is the reward. It is always a valid distribution, and no token is pushed to exactly 0. For a token that is raised, at the stationary point is not more than . It is also limited by the normalization of the probabilities, so it is not more than . Suppose that every generator version is trained to its stationary point and that the current trainer then becomes the new generator. Chaining the rounds together gives , which has the same form as mirror descent. Also, uses the samples of the generator, yet its stationary point corresponds to a regularization in the direction , and the whole process needs no importance weight.
For , the stationary point has no simple closed form. By the analysis of Section 6.1, the pull of this form on a token that is being pushed down is at most about , and the lower the token, the weaker the pull. Roughly speaking, when is below about , the token is pushed all the way to 0. The exact threshold is , where is the value of at the stationary point, so the threshold is a little lower than . The larger is, the lower this threshold, and the fewer tokens are pushed to 0.
One point here is easy to get confused about. The stationary point of is the optimal solution of the problem that is regularized by , while the stationary point that we get by adding directly is not. The reason is the policy gradient term. Under SFC, the expectation of this term is , which is determined by and does not change with . In the problem , however, the gradient of with respect to the logits is , where . The stationary condition of is that is constant, and the optimality condition of the regularized problem is the same condition, so the two points coincide. When is added directly, the stationary condition is . The two sides are multiplied by different matrices, so the stationary point is no longer that optimal solution.
Without any KL, that is, with , the condition for a stationary point becomes that all are equal, which generally does not hold. Then there is no stationary point, and the probability keeps moving to the token with the highest advantage.
Below, two examples compare the three forms side by side. The numbers come from starting at and applying the expected update to the logits again and again. For tokens that are pushed toward 0, the numbers are the limits of the updates.
In the first example, some tokens have a low advantage. Let , and let the advantages of the three tokens be . Then and .
- With , is of the same order as the differences between advantages. stops at , and stops at , so the two are close. The limit of is , where the third token has been pushed to 0. If is increased to 3, stops at , and the third token keeps a nonzero probability.
- With , is smaller than the differences between advantages. The linear stationary point of is . This is not a valid distribution, and the actual updates finally approach . also approaches . stops at . The second and third tokens are pushed very low, but neither of them reaches 0.
In the second example, a token that is rarely sampled gets a higher advantage. Let , and , so is much smaller than the differences between advantages.
- stops at . The probability of the third token grows to about 11 times its original value.
- stops at . The third token takes almost all of the probability. Its is about 100, which is already the upper bound that the normalization of the probabilities allows.
- stops at . The probability of the third token grows to about 5.8 times its original value.
Figure 5 plots the stationary points of the two examples. The buttons at the top switch between the examples. Drag the slider to change from 0.01 to 10, and see how the stationary points of the three forms change with . When is large, the stationary points of all three forms stay close to the generator. The smaller is, the more clearly they separate.
Together, the two examples show that the three forms have different roles. In the first example, when is small, only keeps the tokens with a low advantage, and under the other two forms these tokens are pushed to 0. In the second example, the rare token with a higher advantage grows the least under , and somewhat more under . Under it takes almost all of the probability.
Which behavior we want depends on how reliable the advantage is. If the high advantage of the rare token in the second example reflects a behavior that is really better, letting the token grow fast is good. If the high advantage comes from a stale sample, a flaw in the reward, or plain noise, limiting its growth is safer. The same holds for the first example. If a token with a low advantage is really bad, pushing it to 0 is fine. If it was only unlucky this time, pushing it to 0 loses exploration.
This simplified model also has a limited scope. In real training, each generator version is trained for only a limited number of steps, the generator keeps being refreshed, and different positions share parameters. The stationary points show only in which direction each form pulls, and how far it can pull at most. They cannot predict the actual outcome of training.
6.3 What does each form cost?
Sections 6.1 and 6.2 discussed how each form behaves in expectation. In practice, we also need to consider the variance of each form, and the risks that its behavior brings.
Section 5.1 showed that all three forms can be implemented by modifying the advantage, using only the and the of the sampled token, so they differ little in implementation. Their difference in variance comes mainly from the size of the coefficients. The coefficient of contains . Consider a token that has a small probability under the generator, but whose probability under the trainer has risen. Once this token is sampled, its coefficient is large. The coefficient of grows faster than , which is the same kind of problem as the variance of importance sampling. The coefficient of grows only logarithmically, so its variance is probably the smallest.
Another way to lower the variance is to compute the expectation exactly on the head and to estimate only the tail. In principle, all three forms can do this. Take as an example, and write for the total probability of the head under the generator. For the tokens in the head, can be computed exactly. For a token sampled from the tail, the estimate is . Suppose that we also skip the sampling on the tail, compute exactly only on the head, and merge the tail into a single category that uses only the total probability of the tail. Then the variance that comes from sampling is gone, but the differences inside the tail are lost, so the result is biased. KLPO calls this TopK-KL. Our earlier experiments used this kind of implementation by default. The generator sent back its 128 most probable tokens, and the tail was not sampled. The total probability of the tail was set to the value that the generator gives, and it was split among the tokens of the tail in proportion to the probabilities that the trainer gives them. This is the same as merging the tail into a single category. Once the tail probability is split in the proportions of the trainer, every token in the tail has the same , which equals the total probability of the tail under the trainer divided by the total probability of the tail under the generator. The KL inside the tail then reduces to a single term.
Apart from variance, the asymmetry of itself has a good side and a bad side. The good side is that prevents the probability of a token from being pushed to 0 within one generator version. This avoids a drop in entropy and a loss of exploration, and it still lets tokens that are really better grow fast. The bad side is that the constraint of on the raising side is weak. If the high advantage of a token comes from a stale sample or from noise, also does little to stop the token from growing. Our earlier experiments give one piece of indirect evidence. One method there multiplies the SFC gradient by the ratio truncated at 2, , which is truncated importance sampling. This is the combination that Marek and Ryabinin (2026) use. When this method collapsed on Countdown, the entropy kept rising, which means that tokens that originally had a low probability were raised as a whole. The responses also became longer and longer, and at the end the KL between the trainer and the generator suddenly rose by two orders of magnitude. This looks more like a problem on the raising side. But that run had no KL term at all, so it is only indirect evidence.
The risk of and is on the other side. Sections 6.1 and 6.2 showed that the protection these two forms give to tokens that are being pushed down is limited. When is small, tokens with a low advantage may be pushed all the way to 0.
6.4 How should one choose in practice?
Before comparing the forms, it helps to judge whether the choice matters in your own training. As Section 5.2 said, when is clearly larger than the differences between advantages, the KL keeps near 1, the forms differ little, and the choice has a limited effect. The choice matters only when is of the same order as the differences between advantages or smaller, for example for the token in Section 6.1 with age 50 and a of only 0.042.
If the choice does matter, the next question is which kind of token you worry about more in your training. Suppose you observe that the entropy keeps falling and that some tokens are pushed down until they almost never appear. Then a form that protects the tokens that are being pushed down, such as , may be more suitable. Suppose instead you observe that the entropy rises, that the responses get longer and longer, and that the distribution of becomes very heavy-tailed. Then a form that limits the tokens that are being raised, such as or , may be more suitable. Which kind of token carries more risk may differ across models, data, optimizers and distributions of the age, so this judgment is best based on what you observe in your own training.
When you need to protect against both kinds of risk, you can combine the forms. For example, add both and . By the results of Section 6.2, protects the tokens that are being pushed down, and gives the strongest constraint on the tokens that are being raised, so the two complement each other. The price is that the weights of the two terms have to be chosen separately.
Variance also affects the choice. The coefficient of grows only logarithmically, so its variance is probably the smallest of the three. The coefficient of contains , and the coefficient of grows even faster. With either of these two forms, when a token with a very large happens to be sampled, it brings a very large gradient. The forms also have different requirements for numerical precision, which is the topic of the next section.
Another thing that can be tuned is how depends on the age. Our earlier approach let decay with the age, on the grounds that an older generator is less suitable as the distribution that the current trainer is pulled toward. One can also argue the other way. The older a sample is, the farther it is from the current model and the more it needs a constraint, so should grow with the age. With the numbers of Section 6.1, is only 0.042 at age 50, and in the simplified model, can no longer stop tokens with from being pushed down. Which choice is more suitable depends on which kind of risk the samples of old generators are more likely to bring in training. For tokens with age 0, we set to 0 in our earlier experiments, for a reason related to numerical precision that Section 7.2 explains.
Finally, a KL between the trainer and the generator can only limit how much the model is updated within each generator version. The generator keeps following the trainer, so the distribution that this KL compares the model with also keeps moving. This KL therefore cannot prevent the model from moving farther and farther from the initial model over many rounds. If the long-term drift needs to be limited, one can add a separate KL that compares the model with . The two KL terms complement each other.
7. In finite precision, how should the KL and its coefficients be computed?
TL;DR: To compute the KL and its coefficients, first compute the difference of log probabilities in FP32, and then get with
expm1(rho). To log the value of , useexpm1(rho) - rhoor a Taylor expansion. Do not first compute , which is close to 1, and then subtract 1, because the logged KL can then be negative. If is stored in BF16, is also 0 for every token whose probability rose only slightly, with below about 0.0039, so these tokens get no constraint at all. To check whether the trainer and the generator agree exactly, require that the maximum of equals 0, and do not look at the mean of .
The analysis so far assumes that the computation is exact. In real training, the KL and its coefficients are computed in finite precision. This section shows that the same formula, computed in different ways, can give very different results, and that the different coefficients of Section 5 have different requirements for precision. All numbers in this section were computed with the rounding rules of FP32 or BF16, and the curves in Figures 6 and 7 were computed point by point with the same rules.
The cause of the problem is that floating-point numbers are dense near 0 and sparse near 1. In FP32, the gap between 1 and the next larger number is , while near 0, FP32 can represent numbers as small as order . BF16 is coarser. The gap between neighboring numbers just above 1 is , and the gap just below 1 is . If the result we want is a very small quantity, the computation should avoid an intermediate value of the form 1 plus this small quantity. Once such an intermediate value is stored as a floating-point number near 1, most of the information in the small quantity is lost, and subtracting 1 afterwards cannot bring it back. The next two sections show how this problem appears when we log the value of the KL and when we compute the coefficients used in training.
7.1 Why can the logged value of the KL be negative?
Take as an example, where . Computing it directly by this formula in FP32 gives at . The true value is about . A quantity that is mathematically non-negative comes out negative, and the error is five orders of magnitude larger than the true value. The reason is that is stored as the FP32 number nearest to it, which lies near 1, and the rounding error of this step is of order . After 1 is subtracted, almost all that remains is rounding error. If we compute expm1(rho) - rho instead, expm1 computes directly, without the intermediate value that is close to 1. The result is , which differs from the true value by only 9%. The remaining error comes from rounding the output of expm1. That output has a relative error of order , which is an absolute error of about . So the smaller is, the larger the relative error of . To keep the precision at even smaller , one can compute with the Taylor expansion directly when is very small. This removes the subtraction step completely.
Figure 6 plots, on a logarithmic scale, the true value of and the errors of the three ways of computing it. The error of the direct computation stays at order to , whatever the size of . When is below about , this error is already larger than the true value, and the computed result is meaningless. With expm1, the error is proportional to , and it becomes comparable to the true value only when is below about . With the Taylor expansion, the error stays far below the true value while is small. Only when becomes large do the dropped higher-order terms start to matter.
The same cause can also make a consistency check reach a wrong conclusion. Some checks use a quantity that is written as 1 plus a small second-order quantity, for example . When the samples come from the generator, the expectation of this quantity is 1 plus the sum of the KL in the two directions. In FP32, as long as , this quantity is rounded to exactly 1, which looks as if the trainer and the generator agreed exactly.
The mean of is also not suitable for checking whether the two agree. can be positive or negative, so positive and negative values can cancel, and the mean can be close to 0 or even negative. To confirm that the trainer and the generator agree exactly, a more reliable approach is to compute directly and to require that its maximum is exactly 0. is never negative, so a maximum of 0 means that the trainer and the generator agree exactly on every token.
7.2 How much does precision affect the coefficients used in training?
The coefficients used in training are affected by the same problem, but to different degrees. The coefficient of is . As long as is computed by subtracting FP32 log probabilities, its relative error stays within 0.2%, even if it is converted to BF16 afterwards. But if the log probabilities themselves are stored in BF16, the subtraction loses most of the information. For example, and both become when they are stored in BF16. Their difference is then 0, while the true value is 0.003.
The coefficient of has one more problem. If is computed first and 1 is subtracted afterwards, we run into the situation described at the start of Section 7. Suppose that is stored in BF16. Then at and at , is stored as exactly 1, and is 0. At , becomes 0.0078, while the true value is 0.0050. At , becomes , while the true value is . Because the gaps above and below 1 differ, the errors on the two sides are not symmetric. Every token whose probability rose only slightly, with below about 0.0039, gets the coefficient 0, so it is not constrained at all. A token with a slight decrease is pulled back as soon as its is below about , and the pull is enlarged. Figure 7 plots the steps that we get when we subtract 1 from an stored in BF16, and the two vertical lines mark these two thresholds.
The correct approach is to compute in FP32, to use this directly as , and to get with expm1(rho). The same applies to the estimate on the tail in Section 6.3. The dashed line in Figure 7 shows the result of computing expm1 in FP32 first and storing the result in BF16 afterwards, and it almost coincides with the true value. The coefficient of has no step that subtracts 1. Even if is stored in BF16, the relative error of this coefficient stays within 0.2%, so it is less affected by this problem.
There is another kind of numerical difference that does not depend on the storage precision. Even when the trainer and the generator use the same version of the parameters, the log probabilities that they compute are not exactly the same, because the inference engine and the training framework may use different kernels, precisions, parallelism strategies and batch shapes (Thinking Machines Lab, 2025). Then is not 0, and every form of the KL between the trainer and the generator pulls the trainer toward the numerical error of the inference engine. This is why, in our earlier experiments, we set to 0 for tokens with age 0, that is, tokens for which the trainer and the generator have the same parameter version. Removing this difference completely requires the two sides to compute in exactly the same way, for example with batch-invariant kernels. This is a problem of the inference engine, and it is outside the scope of this article.
8. Closing remarks
This article discussed how the form of a KL relates to the gradient that it produces. When you analyze a line of KL code, it helps to first ask the following questions.
- Which model drew the samples? Was it the current model, an old model, or a fixed generator?
- Where does appear, and which of these places does the code differentiate?
- What is the coefficient that finally multiplies the score, and which direction of the KL does it correspond to?
- In what precision is this coefficient computed?
For the KL that brings the trainer and the generator closer in asynchronous RL, each form has a different role. protects the tokens that are being pushed down. constrains the tokens that are being raised. acts on both kinds of tokens, but its protection of the tokens that are being pushed down is limited. Which form is more suitable depends on which kind of token carries more risk in training, and also on the size of relative to the advantage.
The analysis above covered how these forms behave in expectation, what they cost, and the precision issues in computing them. How they perform in real training, for example whether they can prevent training from collapsing, may also depend on other factors, such as the size of the head, how the tail is handled, and the size of the model. We will discuss these questions with experiments in a separate article.
The coefficients in this article are at the token level. A later article will apply the same idea at the prompt level. For example, when GRPO normalizes the advantage by the standard deviation of the group, this amounts to multiplying each prompt by a weight, and this weight determines which objective the training actually optimizes.
Appendix: How does the naming of KL directions in this article differ from other public sources?
This article follows the naming of our earlier article that adds a KL to Score Function Centering. That article calls the estimator of the sparse reverse-KL estimator, where is the generator and is the trainer. By the same rule, this article calls a KL with in the second place the reverse KL, for example and . A KL with in the first place is called the forward KL, for example and .
Some public sources use the opposite names. RLHF papers and codebases usually view the model being trained, , as the approximating distribution in variational inference. So they call the reverse KL and the forward KL. Articles on on-policy distillation do the same, and they call the KL with the student first and the teacher second the reverse KL. The two naming conventions refer to the same two quantities, with the names swapped.
When you read this article next to other sources, the following points are the easiest to confuse.
- When is used as a loss and the samples come from the current model, its gradient equals the gradient of . This article says that it gives the forward KL, while the sources with the opposite naming would say the reverse KL.
- Under the same conditions, the gradient of equals the gradient of . This article says that it gives the reverse KL, while the sources with the opposite naming would say the forward KL.
- InstructGPT-style PPO puts in the reward, and its gradient corresponds to , which is the forward KL in this article. GRPO uses as a loss, and under the approximation that the samples come from the current model, its gradient corresponds to , which is the reverse KL in this article.
- Liu et al. (2025) say that used as a loss is a first-order approximation of the reverse KL. The reverse KL that they mean is , which is the forward KL in this article.
- Sources often say that the reverse KL is mode-seeking. This means that a KL with first allows to keep only part of the values that the other distribution takes. In the naming of this article, this statement is about the forward KL. In Section 6.2, under , tokens with a low enough advantage are pushed all the way to 0, which is an example of this property.
- The KL term in the earlier article is . It is called the reverse KL both in that article and in this one, so its name does not change.
So when you read such material, the most reliable approach is to check the formulas. First see which distribution is in each of the two places of the KL. Then see which distribution the samples come from, and which places the code differentiates. Once these three points are clear, the names no longer cause misunderstanding.
References
- John Schulman. Approximating KL Divergence. 2020.
- Kezhao Liu, Jason Klein Liu, Mingtao Chen, Yiming Liu. Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization. 2025.
- Yunhao Tang, Rémi Munos. On a few pitfalls in KL divergence gradient estimation for RL. 2025.
- Yifan Zhang et al. On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning. 2025.
- Yifan Zhang. On KL-Regularized Policy Optimization. 2026.
- Martin Marek, Max Ryabinin. Score Centering Stabilizes Off-policy Reinforcement Learning. 2026.
- DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. 2025.
- Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024.
- Long Ouyang et al. Training language models to follow instructions with human feedback. 2022.
- Audrey Huang et al. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. 2024.
- Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. 2025.
Comments
@misc{zhang2026klgradient,
author = {Zhang, Yihua},
title = {Why Does an Unbiased KL Estimate Not Give the Right Gradient? From Common KL Estimators to the Trainer-Generator Constraint in Asynchronous RL},
year = {2026},
url = {https://normaluhr.github.io/en/p/kl-gradient}
}