Score Function Centering: Why a Stale Rollout Does Not Have to Be Importance Sampled

This is the second article in the Score Function Centering series. I expect that after the first article you already have a first understanding of what a score function is, why we do Score Function Centering, and what it achieves. In this article I want to give you a completely new perspective on Score Function Centering, on what it solves in async RL and, more importantly, on what it does not solve yet. I also want to break one belief, the belief that a stale rollout has to be corrected onto the current policy with importance sampling, and to offer some new ways of dealing with it.

A note on authorship and novelty: this article is not a summary of existing work and existing understanding. It is the author's own new perspective, understanding and method for Score Function Centering. Like the first article, this one was written entirely by hand by the author, not by AI. Of course, if you are reading the English version, I am sorry, that one was translated by AI. The author also strongly recommends reading the first article on Score Function Centering first, so that the terms used here and the positions taken here are clear.

1. Starting from what Score Function Centering has not solved

In this section we talk about Score Function Centering first, about what it repairs and what it does not repair yet.

Let us do a quick recap. The previous article started from the score function itself and ended at a very concrete problem in async RL: the rollout is produced by the generator $q_\phi$ , but the trainer that actually does the backward pass has already become $p_\theta$ .

The standard score is still a completely correct log-probability gradient:

$s_\theta(a)=\nabla_\theta\log p_\theta(a)=J_\theta(a)-\mathbb E_{v\sim p_\theta}[J_\theta(v)]$

but it is no longer zero-mean under the distribution that actually generated the data, $q_\phi$ :

$\mathbb E_{a\sim q_\phi}[s_\theta(a)]=\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta]\neq 0.$

What Score Function Centering does is this: since the action is sampled from $q_\phi$ , stop centering with $p_\theta$ and center with

$\tilde s_{q,\theta}(a)=J_\theta(a)-\mathbb E_{v\sim q_\phi}[J_\theta(v)].$

So that:

$\mathbb E_{a\sim q_\phi}[\tilde s_{q,\theta}(a)]=0.$

At this point the central problem of the previous article is solved. We have the zero-mean property back, and we have baseline invariance back.

But there is in fact a larger problem that is still not solved:

We did center the score correctly, but what we are learning is still the learning signal under the generator $q_\phi$ . The trainer is already $p_\theta$ . Is that really fine?

Let us first separate the two problems that generator staleness brings.

What Score Function Centering repairs is score drift. The standard score had a nonzero mean under $q_\phi$ :

$d_{q,\theta}:=\mathbb E_{q_\phi}[s_\theta]=\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta].$

So a reward update that has not been centered properly:

$\mathbb E_{q_\phi}[A(a)s_\theta(a)]$

can be split into:

$\mathbb E_{q_\phi}[A(a)s_\theta(a)]=\operatorname{Cov}_{q_\phi}(A,J_\theta)+\bar A_q\,d_{q,\theta},\qquad \bar A_q:=\mathbb E_{q_\phi}[A].$

The problem with the second term is that $\bar A_q$ is only an average advantage. It says nothing about which token is better and which is worse, and yet it directly controls the update along the trainer-generator mismatch direction. The Score Function Centering of the previous article is, in essence, removing exactly this term:

$g_q:=\mathbb E_{q_\phi}[A(a)\tilde s_{q,\theta}(a)]=\mathbb E_{q_\phi}[(A(a)-\bar A_q)J_\theta(a)]=\operatorname{Cov}_{q_\phi}(A,J_\theta).$

This formula matters a lot. It tells us that the update left after SFC cares about one thing only: under the sampler $q_\phi$ that actually generated the data, which actions have the relatively higher advantage, and what their logit direction is.

But note that it is still:

$g_q=\mathbb E_{q_\phi}[(A-\bar A_q)s_\theta].$

It is not the one under the current trainer:

$g_p=\mathbb E_{p_\theta}[A s_\theta].$

So SFC cannot turn $q_\phi$ into $p_\theta$ out of nothing. All it says is this: since the data was generated by $q_\phi$ , let us at least define the learning signal under that distribution cleanly.

2. Importance Sampling: it has always been done this way, but is it right?

2.1 Where importance sampling comes from

The problem that is still not solved is this:

Should we correct this update under $q_\phi$ into an update under $p_\theta$ ?

What does that question mean? Take a somewhat extreme example. Suppose the generator is running a version of the trainer from 10,000 steps ago, which is to say the current trainer has already graduated from university while the generator is still in primary school. For the same question, say the embarrassment of language models, which is bigger, 0.11 or 0.9, the generator writes a pile of analysis, argues that 11 is bigger than 9, and ends at 0.11 > 0.9. To the current trainer that answer is ridiculous, so the probability that the trainer itself samples this answer, the one the generator was most confident about, is close to 0. If you look at the gradient at this moment, you will find that the answer is wildly wrong and its reward, and therefore its advantage, is very low. Normally the gradient from such a sample should bring a fairly large correction, but it may not only be meaningless to the trainer, it may also overcorrect. Applying that gradient to the trainer directly is like forcing a university student to spend a night studying a wrong exercise that says 1+1=3. The student will probably never make that mistake again, but they may also go mad. This is why importance sampling exists. It tells you how important the sample drawn back then really is to the current version of the model, whatever score it received.

If you have studied a little probability, your first reaction after reading the problem above is most likely importance sampling. Suppose what we really want to compute is:

$\mathbb E_{a\sim p_\theta}[f(a)].$

Write the expectation out:

$\mathbb E_{a\sim p_\theta}[f(a)]=\sum_a p_\theta(a)f(a).$

But all the samples we have come from the generator:

$a\sim q_\phi.$

What do we do? A very simple multiplication and division:

$\mathbb E_{a\sim p_\theta}[f(a)]=\sum_a q_\phi(a)\frac{p_\theta(a)}{q_\phi(a)}f(a)=\mathbb E_{a\sim q_\phi}\left[\frac{p_\theta(a)}{q_\phi(a)}f(a)\right].$

So we define:

$r(a)=\frac{p_\theta(a)}{q_\phi(a)}.$

This is the importance ratio. The essence of importance sampling is in fact very plain: I can only sample from $q_\phi$ , but if I multiply every sample by $p_\theta/q_\phi$ , in expectation I get the result under $p_\theta$ back.

For the example above, if the answer "0.11 > 0.9" has probability 0.99 under the generator but probability 0.01 under the current version of the trainer, then no matter how positive or how negative the advantage of that rollout is, it is multiplied by a very small weight of 0.01/0.99 and stops being important.

Putting this back into the policy gradient works the same way. The current-policy update is:

$g_p=\mathbb E_{a\sim p_\theta}[A(a)s_\theta(a)].$

If the data comes from $q_\phi$ , full importance weighting gives:

$g_p=\mathbb E_{a\sim q_\phi}\left[\frac{p_\theta(a)}{q_\phi(a)}A(a)s_\theta(a)\right].$

In pure theory this is of course the most standard answer. The data comes from an old policy, but through $p/q$ I reinterpret it as an expectation under the current policy. The ratio everyone knows from PPO, $r_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\mathrm{old}}(a_t|s_t)$ , is the same thing once you trace it back. $\pi_{\mathrm{old}}$ produces the rollout, $\pi_\theta$ is the current policy, and the job of the ratio is distribution correction.

So if our goal is clearly I have to recover the current-policy update, importance sampling is entirely reasonable. The only problem is that once we are in async LLM RL, the technique may become a lot less reasonable.

2.2 Why did we start to dislike the importance ratio?

The first problem is very practical: the ratio easily becomes very large. If some token has probability $q_\phi(a)=10^{-6}$ under the old generator while the trainer has already learned it to $p_\theta(a)=10^{-3}$ , then:

$r(a)=\frac{p_\theta(a)}{q_\phi(a)}=1000.$

This means the token is almost never sampled by the generator, but the one time it does appear, its single gradient can outweigh a thousand others. The expectation derived in theory is fine. It is the variance of the Monte Carlo estimator that becomes very large. A long rollout makes this more visible. The probability ratio of a full trajectory is:

$\frac{p_\theta(y_{1:T})}{q_\phi(y_{1:T})}=\prod_{t=1}^{T}\frac{p_\theta(y_t|y_{<t})}{q_\phi(y_t|y_{<t})}.$

Even if the two policies differ only slightly at every token, the product of these ratios can become extreme. Why PPO clips the ratio comes down to the same point: exact importance weighting is theoretically correct, but used directly for stochastic optimization its variance can be large enough to be useless. So methods in practice usually truncate, for example:

$w_\tau(a)=\min\left\{\frac{p_\theta(a)}{q_\phi(a)},\tau\right\}.$

This is of course a lot more stable, but once you truncate you are no longer doing exact importance sampling. Take a token with $q(v)=0.01$ and $p_\theta(v)=0.10$ . With $\tau=2$ , its probability weight under full importance weighting is $0.10$ , and after truncation it is only $0.01\times 2=0.02$ . That is, only 20% of the update contribution this token should have had is kept.

One case here deserves particular attention: the trainer has already learned some effective behavior, so the probability of that token has risen under $p_\theta$ , while the old generator $q_\phi$ has not caught up. That token then easily lands in the high-ratio region. Clipping may at the same time lower the variance and weaken genuinely valuable new behavior.

Note that I said "may". A large ratio does not automatically mean the token is important, and it certainly does not imply that clipping must hurt training. It only tells us this: truncation picks which updates get weakened by distribution mismatch, and whether task value happens to be positively correlated with that mismatch has to be verified separately. This distinction matters.

Besides the variance, importance sampling has another conflict that is more directly with Score Function Centering itself.

2.3 Importance sampling and Score Function Centering are not good buddies

What Score Function Centering builds is a score centered specifically under $q_\phi$ :

$\tilde s_{q,\theta}(a)=J_\theta(a)-\mathbb E_{q_\phi}[J_\theta].$

It satisfies:

$\mathbb E_{a\sim q_\phi}[\tilde s_{q,\theta}(a)]=0.$

If at this point we change nothing and simply multiply by an importance ratio:

$\mathbb E_{a\sim q_\phi}\left[\frac{p_\theta(a)}{q_\phi(a)}\tilde s_{q,\theta}(a)\right],$

then by the importance sampling identity this expectation becomes:

$\mathbb E_{a\sim q_\phi}\left[\frac{p_\theta(a)}{q_\phi(a)}\tilde s_{q,\theta}(a)\right]=\mathbb E_{a\sim p_\theta}[\tilde s_{q,\theta}(a)]=\mathbb E_{p_\theta}[J_\theta]-\mathbb E_{q_\phi}[J_\theta].$

In general:

$\mathbb E_{p_\theta}[J_\theta]-\mathbb E_{q_\phi}[J_\theta]\neq 0.$

In other words, you just spent all that effort centering the score under $q_\phi$ , and one $p/q$ switches the expectation measure from $q_\phi$ back to $p_\theta$ .

The zero-mean property you went to such lengths to recover through Score Function Centering no longer holds.

So the thing you really cannot do here is this:

You cannot take a score that is already centered under $q_\phi$ and multiply it by $p_\theta/q_\phi$ with nothing else changed.

This does not mean that importance sampling and score centering can never be combined. You can of course derive an importance-weighted centering so that the weighted score is zero-mean under $q_\phi$ again. Some work does exactly this kind of thing, but to buy a self-consistent estimator this way is more like making a compromise in the theory, and the result ends up not accurate enough on either side.

So at this point two different routes have appeared.

The first is the importance sampling route:

$a\sim q_\phi\quad+\quad r(a)=\frac{p_\theta(a)}{q_\phi(a)}\quad+\quad s_\theta(a)\quad\Longrightarrow\quad \text{current-policy update under } p_\theta.$

The intuition behind it is: I want to correct the probability of the old data into the probability of now.

The second is the Score Function Centering route:

$a\sim q_\phi\quad+\quad \tilde s_{q,\theta}(a)=J_\theta(a)-\mathbb E_{q_\phi}[J_\theta]\quad\Longrightarrow\quad \text{sampler-relative update under } q_\phi.$

I accept that the data was produced by the past $q_\phi$ , and I first keep the relative reward signal it expresses under its own distribution.

Next, let us talk about how to walk the second route.

3. If I do not do importance sampling, how can I trust this update?

The question is now very concrete. We have decided to keep the sampler-relative update:

$g_q=\mathbb E_{q_\phi}[(A-\bar A_q)s_\theta],\qquad \bar A_q=\mathbb E_{q_\phi}[A].$

But the trainer we actually care about is already $p_\theta$ . The local update of the current policy is:

$g_p=\mathbb E_{p_\theta}[A s_\theta].$

So how far apart can $g_q$ and $g_p$ be? We first use the zero-mean property of the standard score:

$\mathbb E_{p_\theta}[s_\theta]=0.$

So we can subtract any baseline that does not depend on the action inside $g_p$ . To line it up with $g_q$ , we deliberately subtract the same $\bar A_q$ :

$g_p=\mathbb E_{p_\theta}[A s_\theta]=\mathbb E_{p_\theta}[(A-\bar A_q)s_\theta].$

Now you can see that the two are already very close:

$g_q=\mathbb E_{q_\phi}[(A-\bar A_q)s_\theta],\qquad g_p=\mathbb E_{p_\theta}[(A-\bar A_q)s_\theta].$

Their integrands are identical. The only difference is that one takes the expectation under $q_\phi$ and the other under $p_\theta$ . Subtract them:

$g_q-g_p=\sum_{v\in\mathcal V}\left(q_\phi(v)-p_\theta(v)\right)\left(A(v)-\bar A_q\right)s_\theta(v).$

Every term inside the sum, $(q_\phi(v)-p_\theta(v))\times(A(v)-\bar A_q)s_\theta(v)$ , is one token's contribution to the update mismatch. The first factor tells you how far apart the two policies are on that token, and the second tells you how much that token affects the learning direction.

Suppose that in the local region we care about there is a uniform $B$ such that:

$\left\|\left(A(v)-\bar A_q\right)s_\theta(v)\right\|\le B,\qquad \forall v\in\mathcal V.$

Then by the triangle inequality:

$\|g_q-g_p\|\le\sum_{v\in\mathcal V}|q_\phi(v)-p_\theta(v)|\left\|\left(A(v)-\bar A_q\right)s_\theta(v)\right\|\le B\sum_{v\in\mathcal V}|q_\phi(v)-p_\theta(v)|.$

At this point, controlling the gradient mismatch has become controlling the $L_1$ distance between two distributions. Then by Pinsker's inequality:

$\sum_{v\in\mathcal V}|q_\phi(v)-p_\theta(v)|\le\sqrt{2D_{\mathrm{KL}}(q_\phi\|p_\theta)}.$

So we get:

$\|g_q-g_p\|\le B\sqrt{2D_{\mathrm{KL}}(q_\phi\|p_\theta)}.$

This formula gives us a very important conclusion: we did not force the update under $q_\phi$ back into the update under $p_\theta$ with an importance ratio, but as long as we control $D_{\mathrm{KL}}(q_\phi\|p_\theta)$ we can control the error between the sampler-relative local update and the current-policy local update.

For example, if we can maintain:

$D_{\mathrm{KL}}(q_\phi\|p_\theta)\le\epsilon,$

then we get:

$\|g_q-g_p\|\le B\sqrt{2\epsilon}.$

So this route thinks quite differently from importance sampling. Importance sampling says: I apply a correction to the update directly, hoping to recover $g_p$ . Sampler-relative learning plus a trainer-generator KL says: I keep $g_q$ , but I control the distance between $g_q$ and $g_p$ .

One repairs at the level of the estimator, the other limits, at the level of the distribution, the error that comes from not repairing anything.

This bound should not be read too far either. It is a fixed-prefix local analysis. It holds the current prefix and the conditional advantage function fixed, and it does not handle the fact that the visitation distribution of future prefixes also changes once the policy changes. So the $g_p$ here is not the full on-policy trajectory gradient. Besides, if $B$ itself is very large the bound can be loose. What it gives us is a clear design motivation, not a theorem or a guarantee that long training will converge.

4. Wait, is the KL not pulling the current model back into the past?

It is. If what we add is:

$D_{\mathrm{KL}}(q_\phi\|p_\theta),$

where $q_\phi$ is the old generator and $p_\theta$ is the current trainer, then minimizing this KL does produce a force pulling $p_\theta$ toward $q_\phi$ .

But let us pause here. A KL has a direction. What we write here is $D_{\mathrm{KL}}(q_\phi\|p_\theta)$ , which takes the generator $q_\phi$ as the reference and constrains the current trainer $p_\theta$ . That direction is not chosen at random. It is exactly the same thing as the mismatch direction that Score Function Centering removed earlier.

Because $q_\phi$ is fixed while the trainer updates:

$D_{\mathrm{KL}}(q_\phi\|p_\theta)=\mathbb E_{a\sim q_\phi}\left[\log q_\phi(a)-\log p_\theta(a)\right].$

Taking the gradient with respect to $\theta$ :

$\nabla_\theta D_{\mathrm{KL}}(q_\phi\|p_\theta)=-\mathbb E_{a\sim q_\phi}\left[\nabla_\theta\log p_\theta(a)\right]=-\mathbb E_{q_\phi}[s_\theta].$

And from the previous article we already know:

$\mathbb E_{q_\phi}[s_\theta]=\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta].$

So:

$\nabla_\theta D_{\mathrm{KL}}(q_\phi\|p_\theta)=\mathbb E_{p_\theta}[J_\theta]-\mathbb E_{q_\phi}[J_\theta].$

This is interesting. Before Score Function Centering, the extra term in the reward update was exactly:

$\bar A_q\left(\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta]\right).$

And when we minimize $D_{\mathrm{KL}}(q_\phi\|p_\theta)$ , the update direction it provides is also exactly:

$\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta].$

The only difference is that this time the thing controlling that direction is no longer the inexplicable average advantage $\bar A_q$ , but a KL coefficient $\beta$ that we decide ourselves.

So using $D_{\mathrm{KL}}(q_\phi\|p_\theta)$ here has a very natural reading: SFC first takes the uncontrolled trainer-generator mismatch term out of the reward update, and then the KL adds the same direction back as an independent regularization term.

This is why the SFC plus KL decomposition is so neat. It is not simply "add a KL so the trainer and the generator do not differ too much". It decouples the two things that used to be mixed together inside the reward gradient:

$\mathbb E_{q_\phi}[A s_\theta]=\operatorname{Cov}_{q_\phi}(A,J_\theta)+\bar A_q\left(\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta]\right).$

SFC keeps the first term:

$g_{\mathrm{reward}}=\operatorname{Cov}_{q_\phi}(A,J_\theta),$

while the KL takes care of the second direction on its own:

$g_{\mathrm{KL}}=\beta\left(\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta]\right).$

So in the end:

$g_{\mathrm{total}}=\operatorname{Cov}_{q_\phi}(A,J_\theta)+\beta\left(\mathbb E_{q_\phi}[J_\theta]-\mathbb E_{p_\theta}[J_\theta]\right).$

Now "what to learn" is decided by the reward, and "how far the trainer is allowed to be from the sampler" is decided by $\beta$ .

5. One last question: what if this sampler is already very old?

If the KL is pulling toward the generator in the first place, then the older the generator is, the further back into the past it pulls the model. Is that not a problem?

It is.

So when you really do async RL, the tokens should not share one fixed KL strength. A token that was just generated and a token generated a few dozen optimizer steps ago clearly do not correspond to sampler distributions of the same quality as a reference. Suppose token $t$ was produced at sampler step $\sigma_t$ and the current trainer step is $\tau$ . Define its age:

$e_t=\tau-\sigma_t.$

Here I propose one rule that can be used in practice:

$\beta_t=\begin{cases}0,&e_t=0,\\ \beta_0/\sqrt{e_t},&e_t\ge 1.\end{cases}$

That is, the staler the token, the weaker its KL constraint. The intuition behind this design is that a fresh sampler is still close to the current trainer, so it is a fairly trustworthy local anchor, while a sampler that is already old represents a past policy. We can of course still use the relative reward information it left behind when it produced the rollout, but there is no need to force the current trainer to stay tightly attached to it.

So two things have to be separated here: "which $q_t$ each token uses as its reference" and "how strong the KL weight on that reference should be". These are two different questions. The first is decided by the version that actually generated the token. The second is decided by how stale it is.

Setting $\beta_t=0$ for a fresh token has a simple reason too: if the generator and the trainer are the same version in the first place, there is no need to treat the tiny distribution difference that comes from numerics, the inference kernel or the precision as an extra imitation target.

But one thing has to be stated clearly: $1/\sqrt{e_t}$ is not an answer derived from the KL error bound above. The theory tells us that the smaller $D_{\mathrm{KL}}(q_t\|p_\theta)$ is, the smaller the error bound between the sampler-relative local update and the current-policy local update. It does not tell us that $\beta_t$ has to equal $\beta_0/\sqrt{e_t}$ . The latter is only a design choice that I recommend, and you are free to study a constant coefficient, a batch-level coefficient, other age schedules, or even an adaptive adjustment driven by the observed KL.

There is one more place that easily causes confusion. We said above that the reward gradient no longer uses importance weighting, but you may still see $p_\theta(v)/q_\phi(v)$ inside the sparse reverse-KL estimator. There is no contradiction. The role of the importance ratio in the reward gradient is:

$\mathbb E_q\left[\frac pq f\right]=\mathbb E_p[f],$

which changes the distribution under which we learn the reward signal. The ratio inside the sparse KL is there to estimate a completely different target:

$\nabla_\theta D_{\mathrm{KL}}(q\|p_\theta).$

For the logit $z_j$ , for example:

$\frac{\partial D_{\mathrm{KL}}(q\|p_\theta)}{\partial z_j}=p_\theta(j)-q(j).$

If we only sample one $v\sim q$ , we can use:

$\hat g_j(v)=\left(\frac{p_\theta(v)}{q(v)}-1\right)\mathbf 1[j=v],$

because:

$\mathbb E_{v\sim q}[\hat g_j(v)]=q(j)\left(\frac{p_\theta(j)}{q(j)}-1\right)=p_\theta(j)-q(j).$

So the $p/q$ here is only a coefficient inside a sparse Monte Carlo gradient estimator. It does not change the reward update from the $q$ measure to the $p$ measure.

6. So does this method actually work?

Everything above is intuition and derivation, so this section gives some real results.

To check that this method really works, I ran a large number of experiments, and the baselines include SFC and several of its known variants. Space is limited, so this article shows only a small part of them. The full experimental setup, the results, the analysis and the open-sourced code are all in the third article of this series.

The seven settings in the figure below all come from a real asynchronous pipeline: the generator keeps producing rollouts, the trainer keeps updating, and every token carries the version number of its own sampler, so the staleness here is measured. The tasks are Countdown and GSM8K, the models are Qwen3-0.6B, Qwen3-4B, Qwen3-30B-A3B and GLM-4.5-Air (106B total parameters, 12B active), the context length is at most 4,096 tokens, every run takes 3,200 trainer steps, and the measured mean staleness goes from 46 optimizer steps all the way to 367. The reward of both tasks is only 0 or 1. Countdown, for instance, asks the model to write an arithmetic expression over the given numbers whose result equals the target, 1 point when it is right and 0 when it is wrong, so the reward in the figure can be read directly as a solve rate.

The "4-prompt slice" in the figure is the granularity at which the trainer consumes a rollout. The generator produces the rollouts of 64 prompts at a time, the trainer cuts that batch into slices, and every slice takes one optimizer step. A 4-prompt slice means every step uses 4 prompts, and a 1-prompt slice means every step uses only 1. The finer the cut, the more steps one batch of rollouts has to be consumed over, and the older the sampler is relative to the trainer, so a 1-prompt setting is far staler than a 4-prompt one. Every setting in the figure is labelled with the mean staleness that was actually measured for it.

To line up with the discussion above, only four methods are shown: SFC, which does centering alone, SFC+TIS, SFC+MIS, and this article's SFC+KL( $\beta_t$ ) with $\beta_0 = 0.3$ . SFC+TIS multiplies the centered score by a truncated importance ratio $\min(p_\theta(a)/q_\phi(a),\ 2)$ , which is what truncated importance sampling does. SFC+MIS keeps only the tokens whose ratio falls inside $[0.5,\ 5]$ and masks the rest, which comes from IcePop. Combining these two ratio corrections with score centering is what the work mentioned earlier does.

Each bar is the best held-out evaluation reward that the run reached during training. SFC+KL is the only method among these seven settings that trained to the end every time. In five of the seven settings, its best held-out evaluation reward is also the highest of the four methods.

Back to the question this article started from: when you learn from a stale rollout, do you have to "correct it into the present" before you can use it?

The answer is that you do not have to.

Importance sampling answers: the data comes from $q_\phi$ , but I will try to recover the current-policy expectation through $p_\theta/q_\phi$ . Sampler-relative learning answers: since the data was produced by $q_\phi$ , I will first keep the relative reward signal it expresses under its own distribution, and as for how far it has already drifted from the current trainer, I control that with a separate KL constraint.

Put simply, the method proposed in this article drops importance sampling, and it can be seen as the combination of Score Function Centering, a sparse KL and staleness scaling, each with its own job:

SFC: I accept that the data comes from $q_\phi$ , so I center under $q_\phi$ .
KL: but I do not allow $q_\phi$ and $p_\theta$ to drift too far apart.
Staleness scaling: and the older $q_\phi$ is, the less I think its reference value counts.

I think this is a new way of understanding the trainer-generator mismatch in async RL: do not assume that stale data has to be reweighted into current-policy data first. You can also keep the learning signal it carries, and then split "what to learn" and "how far from the current policy" into two separate questions.