Everything You Need to Know About Trainer-Generator Mismatch: Sources, Monitoring, Zero-Difference Tests, and Debugging

Published
18,013words
18equations
en / zhlanguages

In PPO or GRPO, we usually have a natural expectation. We have just sampled a batch of data and have not updated the parameters yet, so the new policy and the old policy should be the same, and the importance ratio should equal 1. In LLM training, this expectation does not always hold. Even when the generator and the trainer load the same weights, the token log-prob that the generator reports during generation, that is, the natural logarithm of the probability, can differ from the value that the trainer recomputes. If this difference is large enough, some tokens are already outside the clipping range before the first update has even started.

When we introduced Score Function Centering, we discussed the distribution difference that appears when the parameter version of the generator lags behind the trainer. However, matching parameter versions does not necessarily mean that the two sides actually compute the same policy distribution. If the computations on the two sides differ, then the KL, the ratio, and the clip fraction that we observe later are affected by both the parameter updates and the execution process.

In this article, trainer-generator mismatch means these differences between the training side and the generation side. If you are integrating an RL training framework, or if you have seen the ratio move away from 1 before the first update, or seen an occasional KL Spike during training, you can use the discussion below to check your own system. To read this article, you only need to know the basic objective of PPO or GRPO. We will write out each formula again when we use it.

This article first discusses how differences between the trainer and the generator affect model updates. Next, we analyze where these differences come from, and what evidence to retain when monitoring RL training, to help with debugging. With this evidence, we explain how a zero-difference test can narrow the investigation. We then use an example in which generation pauses and later continues, to show how to trace a KL Spike to a specific state error. We also discuss temperature transforms and candidate verification, because matching probabilities on both sides do not guarantee that the actual sampling is correct. Finally, we put these checks, together with weight synchronization and regression validation, into one complete debugging workflow. At the end of the article there is an interactive diagram. It helps you find the next check to run, organized by what happens inside the trainer, inside the generator, and between the two sides.

1. Computation differences between the two sides use up part of the PPO update range in advance

TL;DR: When the ratio uses the probability recorded during generation as its denominator, execution differences between the two sides affect the PPO gradient and can cause clipping to remove some tokens’ PPO gradient contributions before the first parameter update. Even if the initial ratio is 1 when the denominator is an old log-prob recomputed by the trainer, this does not establish agreement between the trainer and generator.

We write the prompt as xx and the response as y1,…,yNy_1,\ldots,y_N. When generating yty_t, the prefix that the model has already seen is ht=(x,y1,…,yt−1)h_t=(x,y_1,\ldots,y_{t-1}), which does not include yty_t itself. For this prefix, we write the distribution given by the trainer as πθ\pi_\theta, and the distribution that the generator actually used when it generated this token as πgenerator\pi_{\mathrm{generator}}. This section first considers ordinary token-by-token sampling. The temperature is 1, there is no top-k, top-p, or other logits modification, and we assume that the sampling implementation is correct. Under these conditions, the softmax probability that the generator reports is the actual sampling probability.

If the denominator of the importance ratio uses the probability that the generator actually used when it generated this token, we can first compute the difference between the log-probs of the two sides, and then take the exponential to get the ratio:

ρt=log⁡πθ(yt∣ht)−log⁡πgenerator(yt∣ht),rt=eρt.\rho_t=\log\pi_\theta(y_t\mid h_t)-\log\pi_{\mathrm{generator}}(y_t\mid h_t),\qquad r_t=e^{\rho_t}.(1)

When ρt>0\rho_t>0, the trainer gives this token a higher probability. When ρt<0\rho_t<0, the trainer gives it a lower probability. Here ρt\rho_t is both the log-prob difference between the two sides and the logarithm of the ratio. So any difference in the forward computations of the two sides enters the PPO objective through the ratio. PPO itself cannot know whether this part of the change comes from a parameter update or from numerical computation.

1.1 Clipping cannot tell a policy update from a numerical shift

Recall the per-token objective that PPO maximizes:

Jt(θ)=min⁡ ⁣(rtAt,  clip⁡(rt,1−ϵ,1+ϵ)At).J_t(\theta)=\min\!\left(r_tA_t,\; \operatorname{clip}(r_t,1-\epsilon,1+\epsilon)A_t\right).(2)

Here AtA_t is the advantage, which measures how much better this action is than a baseline, and we treat it as a constant during optimization. ϵ\epsilon sets the clipping range of the ratio. In the range where no clipping happens, the generator probability does not take part in differentiation, so ∇θrt=rt∇θlog⁡πθ(yt∣ht)\nabla_\theta r_t=r_t\nabla_\theta\log\pi_\theta(y_t\mid h_t). The gradient of the PPO objective for this token is therefore

∇θJt=eρtAt∇θlog⁡πθ(yt∣ht).\nabla_\theta J_t =e^{\rho_t}A_t\nabla_\theta\log\pi_\theta(y_t\mid h_t).(3)

In other words, in the unclipped objective, eρte^{\rho_t} directly scales the update coefficient of this token. If ρt\rho_t is a large positive number, this weight is large. If ρt\rho_t is a negative number with a large absolute value, the weight is close to zero. Whether PPO clips the corresponding objective, however, also depends on the sign of the advantage.

If At>0A_t>0, we want to increase the probability of this token. But when rt>1+ϵr_t>1+\epsilon, the PPO objective becomes the constant (1+ϵ)At(1+\epsilon)A_t, so clipping removes this token’s PPO gradient contribution. If At<0A_t<0, the corresponding clipping happens when rt<1−ϵr_t<1-\epsilon. There is one more case to note. When the advantage is negative and the ratio is large, min⁡\min still selects the more negative rtAtr_tA_t, so the upper clipping bound does not limit this coefficient. Everything above concerns the PPO term of the current token. Other tokens or regularization terms can still update the shared parameters.

We can look at a concrete example with ϵ=0.2\epsilon=0.2. When the advantage is positive, clipping removes this token’s PPO gradient contribution when ρt>log⁡1.2≈0.1823\rho_t>\log1.2\approx0.1823. When the advantage is negative, the corresponding condition is ρt<log⁡0.8≈−0.2231\rho_t<\log0.8\approx-0.2231. Suppose that a token has a positive advantage, and that the two sides already differ by ρt=0.25\rho_t=0.25 under the same weight version. Then

rt=e0.25≈1.284>1.2.r_t=e^{0.25}\approx1.284>1.2.(4)

This means that although the trainer has not made any parameter update, clipping already removes this token’s PPO gradient contribution in the first backward pass. So when we check samples from the same version, we need to count how many tokens cross these boundaries, and we also need to record the signs of their advantages. If we only count the fraction of ratios outside the range, we cannot directly know how many of those tokens actually have their PPO gradient contributions removed by clipping.

Even when the difference is not large enough to push the ratio out of range, it still affects the room left for later updates. Suppose the initial ratio is already 1.11.1. Then the trainer only needs to raise the probability by about 9.1%9.1\% for the total ratio to reach 1.21.2. If the initial ratio is 1, the probability has to rise by 20%20\% to reach the same upper bound. Of course, if the initial ratio is below 1, there is more room for an upward update. In either case, the change that clipping limits already includes the existing difference between the trainer and the generator.

1.2 Why the old log-prob recomputed by the trainer hides this problem from the metrics

Some implementations let the trainer recompute the log-probs of a batch before the update, and save the results as a fixed reference for the later updates. We write this pre-update trainer distribution as πold\pi_{\mathrm{old}}, and then use only πθ/πold\pi_\theta/\pi_{\mathrm{old}} for PPO clipping. For now, assume that the weights the generator loaded when it generated the data are the same as the weights of the trainer before the update. Then we can split the original ratio into two parts. One part compares the probabilities that the trainer gives before and after the parameter update. The other part compares the probabilities that the trainer and the generator give before any update. All three probabilities below refer to the same prefix hth_t and the same token yty_t:

πθπgenerator=πθπold⏟same engine, before and after the update  πoldπgenerator⏟same weights, two engines.\frac{\pi_\theta}{\pi_{\mathrm{generator}}} =\underbrace{\frac{\pi_\theta}{\pi_{\mathrm{old}}}}_{\text{same engine, before and after the update}} \;\underbrace{\frac{\pi_{\mathrm{old}}}{\pi_{\mathrm{generator}}}}_{\text{same weights, two engines}}.(5)

For example, suppose the generator gives a probability of 0.200.20 when it generates some token. After this batch has been sampled, and before the first update, the probability that the trainer recomputes is 0.220.22. The second ratio is then already 1.11.1, and it reflects the computation difference between the two sides under the same weights. Next, if a parameter update raises the trainer probability from 0.220.22 to 0.2310.231, the first ratio becomes 1.051.05, which reflects the change caused by this update. Relative to the actual sampling probability, the total ratio is 1.05×1.1=1.1551.05\times1.1=1.155. If we look only at the first factor, we cannot see the 10%10\% difference at the starting point.

Suppose we run the test when this batch has just started to be used for training, so the parameters have not been updated yet, and the numerator and the denominator reuse the same trainer score. Then the first ratio πθ/πold\pi_\theta/\pi_{\mathrm{old}} is indeed 1. The second ratio πold/πgenerator\pi_{\mathrm{old}}/\pi_{\mathrm{generator}}, however, is not necessarily 1, because the data still come from the generator, and the two engines can give different probabilities. This split lets clipping limit only the size of the trainer update, but it does not automatically remove the difference between the data-generating distribution and the trainer distribution. If we ignore the second factor, the gradient that a single token gets at this starting point is At∇θlog⁡πθ(yt∣ht)A_t\nabla_\theta\log\pi_\theta(y_t\mid h_t). In other words, although the sample actually comes from the generator, we give it the same weight as a sample that the trainer drew itself. In addition, if the trainer recomputes the numerator and this recomputation changes the batch or the numerical path, the first ratio can also move away from 1 because of execution differences, even before any parameter update.

For a fixed prefix and fixed parameters, let f(yt)f(y_t) be a quantity whose expectation we want to compute. Full importance weighting then corrects the expectation over the next token: Eπgenerator[rtf(yt)]=Eπθ[f(yt)]\mathbb E_{\pi_{\mathrm{generator}}}[r_t f(y_t)]=\mathbb E_{\pi_\theta}[f(y_t)]. This equation has two preconditions. The denominator must really describe the actual sampling distribution. Also, every token with nonzero probability under the target distribution must have a chance to appear under the sampling distribution. The second condition is usually called support coverage. In addition, this only corrects the token distribution given the prefix. It does not correct the distribution of the prefixes themselves, because the prefixes are still generated by the generator. In practice, training can use the full weight, or it can control variance with truncation or with another objective. Whichever choice we make, if we observe only the first ratio, we cannot know how large the execution difference is that these algorithms have to handle.

The approach of keeping the reference that limits updates separate from the data-generating distribution also appears in the study of policy optimization by Hilton et al. Here, πold\pi_{\mathrm{old}} is the trainer distribution that is frozen when the updates on each batch begin. It plays a different role from the reference model, which usually stays fixed through the whole training run. With asynchronous training, the second ratio also includes the effect of the generator parameter version lagging behind. So when we want to check execution differences specifically, we first need to fix both sides to the same weights.

Checking whether the ratio deviates from 1 tells us whether the probabilities on the two sides are the same, but it does not tell us where the difference comes from. To exclude the effects of parameter updates and version lag, we can first fix the weights on both sides, and then compare the log-probs of the same token on the same prefix. If the two sides still give different values under these conditions, we need to go on and check the computation paths that they actually execute. We start from the differences between these paths.

2. With the same weights, why do the trainer and the generator still compute different probabilities?

TL;DR: Identical weights, output dtypes, and backend names do not guarantee identical operation order, intermediate precision, or state. Debugging therefore requires comparing the trainer with the generator and comparing prefill with decode inside the generator, while confirming that every comparison uses the same prefix.

In mathematics, we usually treat a model as a function determined by its parameters. When this function becomes a running program, however, we also have to choose a concrete order of operations, the data types, and a way to use the state that has already been saved. For example, the trainer usually processes a whole sequence at once, while the generator first processes the prompt with prefill and then generates tokens one at a time with decode. During decode, the generator also reads the keys and values that it saved earlier, that is, the KV cache. So although the two sides use the same mathematical model, they do not necessarily execute the same set of floating-point operations.

2.1 Floating-point addition is not associative, and the batch shape can change the reduction order

Take a=108a=10^8, b=−108b=-10^8, and c=1c=1, and round to FP32 after every step. We get

(a+b)+c=1,a+(b+c)=0.(a+b)+c=1,\qquad a+(b+c)=0.(6)

In the first order, the two large numbers cancel first, and then 1 is added, so the result is 1. In the second order, we first compute −108+1-10^8+1. At this magnitude, FP32 cannot represent a difference of 1, so the result is rounded back to −108-10^8. Adding 10810^8 to it then gives 0. The information is lost in an intermediate step, so raising the precision of later computations cannot recover it.

In a real model, a change of batch can change the order of computation in this way. Take matrix multiplication as an example. One output element needs to add many products together. The GPU can let one unit of work accumulate these products in order. It can also split the sum into several segments, compute a partial sum for each segment, and then combine the partial sums. The second approach increases parallelism, but it also changes how the additions are grouped. When the batch is large, the many independent rows already provide enough parallel work. When the batch is small, the kernel may choose to also split the work along the reduction dimension. So even if the input of a request does not change at all, its result can change when the shape of the batch it belongs to changes. The PyTorch documentation on numerical accuracy also states that batched computation and computation on individual items are not guaranteed to give bitwise identical results.

We need to distinguish repeatability under a fixed configuration from the result after the batch changes. A kernel can give the same result every time for a fixed shape, and still give a different result, equally stable, as soon as the batch size changes. Fixing the random seed cannot remove this difference, because the difference already appears in the logits before any random number is used for sampling.

Batch invariance requires that the result for a request does not change with the batch size, with the position of the request in the batch, or with changes in the other requests. For operators that contain a reduction, one way to implement this is to fix the tiling and the reduction order for each output. Then a larger batch only adds more independent work, and does not change how the same output is summed. The article on determinism in LLM inference and the open-source operators from Thinking Machines provide concrete implementations. This has a cost, of course. Some performance optimizations that choose a strategy based on the shape become restricted, especially when the batch is small.

However, even if both kernels satisfy batch invariance, they can use two different fixed summation orders. Then the result on each side is stable, but the two sides still differ from each other. So to remove trainer-generator mismatch, we also need the two sides to use numerical paths that match each other.

2.2 The same final dtype does not mean the same intermediate computation

Besides the order of operations, the step at which rounding happens also affects the result. Suppose two adjacent operations are ff and gg, and RR denotes rounding a result to some lower precision. One path stores the intermediate result at low precision before the next step. Another, fused path keeps a higher-precision intermediate value and rounds only at the end. The two paths then actually compute

R(g(R(f(x))))andR(g(f(x))).R\bigl(g(R(f(x)))\bigr) \quad\text{and}\quad R\bigl(g(f(x))\bigr).(7)

Even if the final output dtype is the same, these two expressions are not necessarily equal. So confirming only that "both models are BF16" is not enough to decide whether the numerical paths match. We also need to know whether the weights were converted when they were loaded, which precision the multiplications and the accumulators use, and when intermediate tensors are written back. For example, with LoRA, we can merge the weights first and then multiply, or we can compute the base model output and the adapter output separately and add them at the end. These two approaches can round at different places.

The KV cache also carries earlier numerical differences into later computations. This is because decode uses the K and V that were written into the cache earlier, while the trainer recomputes them for the full sequence. If the storage precision of the cache differs from the computation path of the trainer, later attention steps keep reusing values that have already been rounded. In addition, the cache may belong to the wrong parameter version. For example, if the old cache is still used after a weight update, the old K and V no longer represent how the current model computes this prefix. These two kinds of problems need to be handled separately. For differences caused by rounding, we check the numerical precision of the cache and of the computation. For a cache that still comes from an old version, we check how the existing state is handled when the weights are updated.

For MoE models, we also need to consider the discrete choice of experts. When two router scores are very close, a small numerical change upstream can make top-k select a different expert. The function that is actually executed afterwards then changes too. So if we see the difference grow suddenly after some layer, we need to compare the input that this layer receives, the router scores, and the experts that were finally selected. When we discussed GSPO and the training stability of MoE models, we mentioned Routing Replay. It records the expert indices that each token selects in each layer, and lets later forward computations reuse these choices.

However, which side the replayed expert indices come from determines which kind of difference replay can constrain. The paper "Stabilizing Reinforcement Learning with LLMs" distinguishes two approaches. The first is Vanilla Routing Replay, or R2. It saves the experts that the trainer selected when it recomputed the log-probs before the update, and the later updates keep using these indices. So it constrains the change in πθ/πold\pi_\theta/\pi_{\mathrm{old}} that comes from routing changes. If the trainer and the generator already select different experts before any parameter update, R2 cannot make the choices on the two sides agree.

The other approach is Rollout Routing Replay, or R3. It saves the expert indices that the generator actually selected during generation, and lets the trainer reuse these choices both when it recomputes the old log-probs and during the later updates. So R3 targets the routing difference between the two sides more directly. However, even when the expert indices are the same, the mixing weights and the internal computations of the experts can still differ, so R3 alone cannot guarantee that the log-probs on the two sides are equal. Moreover, even before the first update, forcing the trainer to use the choices of the generator can change the policy that the trainer would otherwise get from its own routing. So when debugging, we can use replay as one controlled comparison that isolates routing differences. If we plan to train with it over the long term, we also need to evaluate how this constraint affects the optimization objective.

2.3 Prefill and decode in the same generator can also execute different computation paths

The differences discussed above can also appear inside the same generator. Prefill processes many tokens at once, while decode usually processes only one new token for each request, so the two stages differ in matrix shapes and in how they read and write the cache. To optimize each of the two stages, the same inference engine may also choose a different algorithm for each stage. For example, the SGLang documentation on attention backends lets users choose the prefill backend and the decode backend separately, and it also explains that a top-level backend can call different sub-implementations.

Besides choosing different kernels, the two stages may also use different forms of the computation. For example, Multi-Head Latent Attention, or MLA, in DeepSeek-V2 can absorb the key projection matrix into the computation on the query side. Consider only the content part, without positional encoding. We write the query vector as q\mathbf q and the compressed vector in the cache as c\mathbf c, and let WW be the projection matrix that recovers a key from the compressed vector. The same attention score can then be written in the following two forms:

q⊤(Wc)=(W⊤q)⊤c.\mathbf q^\top(W\mathbf c)=(W^\top\mathbf q)^\top\mathbf c.(8)

The left side first recovers the key, while the right side first transforms the query, so the right side does not need to explicitly recover a key for every past position. The two forms are equal in real arithmetic. However, they produce different intermediate tensors, use different multiply-add orders, and round at different places, so their floating-point results are not necessarily equal. So if prefill and decode use these two forms respectively, we still need to check whether their numerical results agree, even if the weights and the final dtype are exactly the same. To illustrate this reordering of the computation, we discussed only the content part here. The full MLA also needs to handle positional encoding.

The MoE routing divergence mentioned earlier can also turn a tiny numerical difference into a different computation branch. To see how this affects the output probabilities, we can construct an example with a top-1 router. Two experts output the scalars +1+1 and −1-1 for the current input, and we write the output of whichever expert the router selects as uu. The output layer that follows has only two tokens, with logits [u,0][u,0]. In this case, even if the two paths differ only slightly in the router scores, we can get the following result:

PathScores of the two expertsSelected output uuProbability of the first token
prefill[1.0001,1.0000][1.0001,1.0000]+1+1e/(1+e)≈0.7311e/(1+e)\approx0.7311
decode[0.9999,1.0000][0.9999,1.0000]−1-11/(1+e)≈0.26891/(1+e)\approx0.2689

In this example, the router scores change by only 0.00020.0002, but the log-prob of the same output token already differs by log⁡(0.7311…/0.2689…)=1\log(0.7311\ldots/0.2689\ldots)=1. We set the weight of the selected expert to 1 so that we can observe the effect of the discrete choice alone. In a real model, the size of the difference also depends on the expert outputs, the mixing weights, and the later layers. This example shows that even when the final probability difference is large, the initial numerical difference can be very small. So when debugging, we need to keep searching earlier in the computation, to see what changed before the first routing divergence. Of course, the fact that a routing difference and a probability difference appear together does not prove a causal relation between them. We still need another controlled comparison that fixes the routing on the same input.

So even if prefill and decode are configured with an attention backend of the same name, we still need to check the kernels they actually call, the input shapes, and the intermediate precision. We also need to check the parts outside attention, namely the router, the expert computation, and the combination of results. If the difference does not change after we switch a backend, all we can confirm so far is that this change did not remove the measured difference. If we have not confirmed that the program actually entered the new computation path, we have even less reason to rule out the cause that we originally suspected.

2.4 The two sides may not even agree on the prefix

Besides numerical computation, there is a more direct kind of cause. The inputs that the two sides receive, or the states that they use, may simply be different. Whether it is the token IDs, the position IDs, the attention mask, or the preprocessing of multimodal inputs, any difference among them can make the two sides compute different conditional distributions. For an autoregressive model, an error at one position can also affect many later tokens, because the state produced at this position continues to take part in later generation.

This kind of prefix misalignment can happen when a response pauses during generation and then continues. Here, a generation request means asking the model to generate one response for one prompt. The generator usually handles many requests at the same time, and a scheduler decides which requests get computation in each round. For example, when the KV cache runs out of space, the scheduler may temporarily stop scheduling further computation for one of the requests. Later, it adds the request back to the batch, and the request continues from the tokens it has already generated. In this article, "request resumption" means this process, in which the same unfinished request takes part in computation again. To continue generation correctly, the engine must keep or rebuild the prefix, the positions, and the cache state that correspond to the existing tokens.

For example, the recorded prompt is xx, the response has already generated A, B, and C, and the next token to generate is D. Suppose that when this request rejoins the batch after the pause, the prefix that the generator uses to continue generation is missing C. Then the generator may actually generate D right after B. When the trainer later recomputes the probability of D from the complete sequence that was finally saved, the two sides are comparing

log⁡πθ(D∣x,A,B,C)andlog⁡πgenerator(D∣x,A,B).\log\pi_\theta(D\mid x,A,B,C) \quad\text{and}\quad \log\pi_{\mathrm{generator}}(D\mid x,A,B).(9)

Even if the two programs use exactly the same weights and operators, we should not expect these two numbers to be equal, because the prefixes they condition on to compute the probability of D are already different. D here is still a valid token, and the program may keep running normally, so checking only whether a token ID is outside the vocabulary range cannot find this problem. The request resumption case later in this article explains in more detail how the output count and the token list become misaligned in this way while C is still on its way back.

We can now see that even after confirming that the two sides use the same weights, we still need to check their order of operations, their numerical precision, and their actual inputs. In real debugging, however, we usually do not know at the start which kind of difference we are facing. So next, we look at what information monitoring should save to help us tell these possible causes apart.

3. The mean KL describes the overall difference, but only abnormal tokens show where the problem is

TL;DR: A small mean KL can still hide a few severe anomalies, and the token with the largest difference may not be where the error began. Monitoring needs to retain the inputs and execution state needed to reproduce the anomaly so that we can trace the sequence back to the first divergence.

Given the prefix hth_t, if the token really comes from πgenerator\pi_{\mathrm{generator}}, and the two sides compare probabilities on the same prefix, then

Eyt∼πgenerator(⋅∣ht)[−ρt]=KL ⁣(πgenerator(⋅∣ht)∥πθ(⋅∣ht)).\mathbb E_{y_t\sim\pi_{\mathrm{generator}}(\cdot\mid h_t)}[-\rho_t] =\mathrm{KL}\!\left(\pi_{\mathrm{generator}}(\cdot\mid h_t)\middle\|\pi_\theta(\cdot\mid h_t)\right).(10)

This equation lets us understand the log-prob difference between the two sides as a difference between distributions. The three forms that we used when discussing KL estimators become k1=−ρtk_1=-\rho_t, k2=ρt2/2k_2=\rho_t^2/2, and k3=eρt−1−ρtk_3=e^{\rho_t}-1-\rho_t here. If the two distributions can produce the same set of tokens and both give these tokens nonzero probability, then the expectations of k1k_1 and k3k_3 equal the KL above, while k2k_2 is a second-order approximation when the two distributions are close. In practical monitoring, we usually average a statistic over the valid generated tokens in the rollouts. The result is an average along these sampled prefixes, and it cannot be used directly as the KL of the full sequence. Below we use mean⁡(ρ2)\operatorname{mean}(\rho^2), the average of 2k22k_2, to see what information is lost when only summary statistics are kept.

3.1 The same mean can come from two very different failures

Suppose we have collected 10,000 tokens, and we construct two groups of differences. In the first group, every token has ∣ρ∣|\rho| equal to 0.20.2. In the second group, 9,999 tokens get exactly the same log-prob on both sides, and only one token has a difference of ρ=−20\rho=-20. If we square the log-prob difference of each token and average over all tokens, both groups give 0.040.04:

0.22=9999×0+20210000=0.04.0.2^2=\frac{9999\times0+20^2}{10000}=0.04.(11)
StatisticEvery token differsOnly one extreme outlier
mean⁡(ρ2)\operatorname{mean}(\rho^2)0.040.04
max⁡∣ρ∣\max\lvert\rho\rvert0.220
99th percentile of ∣ρ∣\lvert\rho\rvert0.20
Number of tokens with a nonzero difference10,0001

In the second group, the ratio of that abnormal token is e−20≈2.06×10−9e^{-20}\approx2.06\times10^{-9}. In other words, the probabilities that the two sides give this token differ by almost nine orders of magnitude. If we look only at mean⁡(ρ2)\operatorname{mean}(\rho^2), the two groups are indistinguishable, and even the p99 of the second group shows nothing abnormal. The two cases also point to different directions of investigation. For the first group, we can first check the execution configuration that all requests share. For the second group, it is more useful to export the complete rollout that contains the abnormal token, and then search along the sequence for the first position where the two sides diverge.

The two groups also affect model updates differently. In the first group, if a token has ρ=+0.2\rho=+0.2 and a positive advantage, clipping removes its PPO gradient contribution because it already exceeds log⁡1.2\log1.2. In contrast, ρ=−0.2\rho=-0.2 corresponds to a ratio of about 0.8190.819, which has not crossed the lower bound. Now consider the extreme token in the second group. If the denominator uses the generator probability, then with a bounded advantage, the coefficient of this token itself is very small, or it has already been clipped. If only the old log-prob recomputed by the trainer is used, however, the original difference disappears from the ratio. Whichever objective we use, this anomaly deserves further investigation, because it may reveal a state error that also affects other requests.

So it is best to record the mean, the maximum, and the number of tokens above a threshold together. They show, respectively, the size of the overall deviation, the size of the rare anomalies, and how many tokens the anomalies affect. The threshold can be set from a normal baseline that is measured with the same version and the same execution configuration. When aggregating across GPUs, we also need to pay attention to how the values are combined. If we take the maximum on each GPU and then average these maxima, the extreme difference on a single GPU gets pushed down. The global maximum should be the maximum of the per-GPU maxima, and the global mean should be the total sum divided by the total number of tokens. The more tokens we observe, the more likely we are to see extreme values, so when we compare the maxima of different runs, we also need to look at their sample sizes.

3.2 The largest difference may appear after the error has already spread

For the few tokens with the largest differences, we need to save their rollout IDs, their sequence positions and token IDs, the log-probs from both sides, and the full token-ID sequence needed to reconstruct the complete input. Saving only the decoded text is not enough here, because text that looks similar can correspond to different token sequences. Without the original token IDs, we may be unable to reconstruct the input that the model actually received. In addition, the weight version, the generator node, the batch shape, and cache-related events should also be saved with the sample. Otherwise, once the original scheduling conditions are gone, the problem may be very hard to reproduce.

After finding the abnormal rollout, we also need to look back along the sequence to confirm where the difference first started. The largest difference may appear at the end of a long repeated passage, while the error that actually caused it happened dozens of tokens earlier. If the difference shows up as a persistent small shift, we can first compare the numerical paths that all requests share. If the difference rises across the board after some synchronization, we can check the synchronized weights and whether the old cache was handled correctly. If large shifts start at some position and continue after it, we can check the inputs, the cache, and the request state before and after this position. These patterns, however, only help us choose a direction for the investigation. To confirm the specific cause, we still need to compare one factor at a time, by fixing the weights, changing the batch, clearing the cache, or checking the state.

There is one more precondition to remember. If the prefix that the generator actually used is already inconsistent with the record, then subtracting the log-probs of the two sides gives the difference between computations under two different conditions, and the KL equation above no longer applies. The request resumption example later in this article shows this situation in detail.

4. Once we have found abnormal tokens, why do we still need a zero-difference test?

TL;DR: By fixing inputs and weights and matching computation paths, a zero-difference test provides a controlled comparison for locating execution differences. Recomputing scores with the generator’s prefill path lets us compare trainer with prefill and prefill with decode separately, then investigate the paths that diverge.

Suppose monitoring has exported an abnormal response. The trainer and the generator have the same weight version, but several tokens have large log-prob differences. From this response alone, we do not know which configuration to check first. The two sides may always use different summation orders, and this input just amplified the difference. It is also possible that a synchronization, a cache reuse, or a request resumption went wrong, and made this response depart from its normal computation path. If we simply raise the alert threshold, both cases may go unnoticed. Lowering the learning rate also does not help us tell which case happened.

To narrow down the cause, we need a set of comparisons whose conditions we can control. Concretely, we fix the inputs and the weights, change only the computation path or the execution event that we want to investigate, and observe whether the difference changes with it. Such a comparison does not require the two sides to be exactly equal at the start. For example, if a large shift suddenly appeared after a request resumption, we can first compare it with a path that does not go through resumption. A zero-difference test is a stricter comparison of this kind. It makes the two sides use matching computation paths, and requires every token to get exactly the same log-prob on both sides. If the two sides still differ, we search along the computation path for the first divergence. If they already agree, we restore the original configuration or execution events one at a time, and observe where the difference reappears.

In some situations, such a zero-difference reference is especially useful. For example, when we replace a model implementation or an inference engine, we need to confirm whether the new path introduces a shift. When we check weight synchronization, the effect of a very small parameter update can be hidden by the existing numerical difference. If the size of an anomaly is already close to the usual numerical deviation, adjusting the alert threshold further can hardly separate the two. A zero-difference configuration lowers the numerical baseline of these checks to zero, so it can expose changes that a tolerance would let through. However, if we already have a clear clue that points to a state error, we can first run the corresponding local comparison. We do not need to wait until the whole model reaches zero difference before we start debugging.

Here, "matching" covers the attention implementation, the precision of computation and of the cache, the parallel layout, and the reduction order of any communication across GPUs. So setting both sides to BF16, or enabling only the batch-invariant kernels of the generator, does not yet satisfy all the conditions. Production training does not have to use this zero-difference configuration either. We can keep a separate diagnostic configuration that runs more slowly and covers less, and use it only for these comparisons. If the numerical paths of the two sides have not been aligned, a nonzero difference does not by itself prove that the program has a bug. Conversely, if the difference becomes zero after we switch the configuration, this only shows that the difference is related to the path that was changed. It does not directly prove that the original path has no error.

In engineering practice, this test is often called "0 KL", but when deciding whether it passes, we should directly compare the raw log-prob of every valid token. The reason is that k1k_1 can be positive or negative, so its values can cancel when averaged. Near zero, k3k_3 involves subtracting nearly equal numbers, so rounding can make it zero or even slightly negative. Comparing the raw log-probs token by token avoids mistaking these cases for a pass. If the goal is bitwise equality, that is, equality in every bit, we also need to check the raw bit patterns in the same dtype. A formatted log that prints zero everywhere is not enough to prove bitwise equality.

4.1 Fix the generated sequence so that both sides score the same prediction event

A minimal test can start with one short prompt and a few short responses. When the generator generates the responses, we need to save the token IDs that were actually generated, the log-prob of every generated token, and the corresponding weight version. Then the trainer runs teacher forcing on each saved sequence. That is, the trainer uses the generated tokens as input and recomputes the probability at each prediction position. In this test, both sides turn off dropout, set the temperature to 1, and use no probability truncation or other logits modification.

What we fix here is the sequence that has already been generated. We cannot let the two programs each generate again and then compare the responses. Once a different token is sampled at any position, all later prefixes differ, and we can no longer measure the computation difference on the same input. For the same reason, fixing the random seed cannot replace saving the actual token sequence.

Take [P1, P2, P3, A, B] as an example. The first three tokens are the prompt, and A and B are the response. The output at the position of P3 predicts A, and the output at the position of A predicts B. So if we use the output at the position of B to compare the probability of B, we are off by one position. For a short sequence without padding, the indexing in PyTorch can be written as follows. Here token_ids has shape [1, L], where LL is the total length, and valid_vocab_size is the size of the valid vocabulary. The code first drops the extra columns that were padded for parallel alignment:

valid_logits = logits[..., :valid_vocab_size]
logp = valid_logits.float().log_softmax(dim=-1)
next_token_logp = logp[:, :-1].gather(
    dim=-1, index=token_ids[:, 1:, None]
).squeeze(-1)
response_logp = next_token_logp[:, prompt_length - 1:]

The first position of next_token_logp corresponds to the second token of the input. So when the prompt length is 3, the log-probs of the response tokens A and B start at index 2. This code mainly illustrates the indexing and the range of normalization. For a short sequence, we can compute the full FP32 log-softmax directly. When we build the zero-difference test, however, we should use the log-prob computation path that the training loss actually uses. If the loss uses a fused cross-entropy kernel, while the check uses a separate log-softmax, the result of the check does not necessarily represent the probabilities that the loss really uses. Which positions to select in the end also depends on padding, EOS, and the valid generation length.

The samples used for the check must also contain every generated token that we originally planned to compare. If we first filter out anomalies by ratio and then check the remaining samples, the errors have already been removed before the comparison begins. So after aligning the positions, we need to check the number of valid tokens, whether the values are finite, and the token-by-token differences together. An empty sample and a missing log-prob should both count as failures. What we check here is the probability of the selected token. If we also want to check the full next-token distribution, we can first compare the complete log-prob vectors on a small vocabulary.

4.2 Recompute with generator prefill to split the difference into two paths

If the comparison between the two sides fails, switching the whole model to higher precision often only tells us that "the difference got smaller". It still does not tell us which part of the computation the difference comes from. To narrow the scope of the investigation, we can add a path in which the generator itself rescores the whole sequence, using the same weights and the same token sequence. For tokens produced during decode, we then have three results to compare: the score of the trainer on the full sequence, the prefill score of the generator on the full sequence, and the decode score that was actually recorded during generation.

Prefill is also a path that we need to check here. Even though it belongs to the same engine as decode, we cannot directly treat the probability it gives as the actual probability at generation time. If the generator already shows a difference when it recomputes the log-probs of a response that it generated itself, then we can reproduce the difference without involving the trainer. Even if we later align the trainer completely with this prefill path, the difference between prefill and decode still remains. So proving only that the two full-sequence scores agree is not enough to cover the computation path that actually produced the training data.

To make this rescoring an independent reference, we need to start from a clean cache. For example, prefix caching in vLLM lets a new request reuse the KV cache of an existing prefix. If this computation happens to reuse the state left behind by the failed run, it can no longer serve as an independent comparison. So we first turn off prefix cache reuse and clear the old state, and then submit the complete token sequence to get the prompt log-probs. If prefill itself is computed in chunks, we also need to record the chunking, because it also affects the actual numerical path.

We write these three log-probs as ℓttrain\ell_t^{\mathrm{train}}, ℓtprefill\ell_t^{\mathrm{prefill}}, and ℓtdecode\ell_t^{\mathrm{decode}}. The first two values are both recomputed on the same recorded prefix, while the last value keeps the result that was actually obtained during generation. By adding and subtracting the prefill score in the middle, we can split the total difference into two parts:

ρt=ℓttrain−ℓtdecode=(ℓttrain−ℓtprefill)+(ℓtprefill−ℓtdecode).\begin{aligned} \rho_t&=\ell_t^{\mathrm{train}}-\ell_t^{\mathrm{decode}}\\ &=(\ell_t^{\mathrm{train}}-\ell_t^{\mathrm{prefill}}) +(\ell_t^{\mathrm{prefill}}-\ell_t^{\mathrm{decode}}). \end{aligned}(12)
trainerRecomputes the recorded sequence in full
Full-sequence implementation differenceWeights and forward computation
generator prefillDoes not reuse the old cache
Incremental path differenceDecode computation or state
actual decodeResult saved during generation
Figure 1: Adding the prefill score splits the original comparison into two parts. The comparison between the trainer and prefill compares two full-sequence computations. The comparison between prefill and decode checks the differences in the incremental computation and in the actual execution state.

With these three results, we can choose the next direction of investigation. For example, if the trainer and generator prefill agree while decode differs, we can focus the investigation on the step between prefill and decode. If the two paths inside the generator agree with each other but both differ from the trainer, we can first check the weight conversion, the operator implementations, and the parallel configuration of the two engines. The post "0 train-infer mismatch for Open-weight MoE RL in Open-source code" also aligns the forward computation along these three paths, and further checks the cache state that the next decode step will read.

We can use constructed data for a single token to see what problem remains when only one of the two parts is aligned. Suppose the log-probs from the trainer, prefill, and decode are −1.6-1.6, −1.7-1.7, and −2.0-2.0. The total difference is then 0.40.4. Of this, the trainer and prefill differ by 0.10.1, and prefill and decode differ by 0.30.3. Even if the first part is aligned completely, the remaining ratio is still e0.3≈1.350e^{0.3}\approx1.350. For a token with ϵ=0.2\epsilon=0.2 and a positive advantage, this already exceeds the clipping upper bound discussed earlier, and there is no effect of parameter version lag here.

What can be added directly here are the per-token log-prob differences, and the means computed over the same set of tokens with the same statistical weights. If we compute k3k_3 separately for the two differences, we can no longer add the results and treat the sum as the k3k_3 of the total difference. The reason is that k3k_3 is a nonlinear function of the difference ρ\rho, namely k3(ρ)=eρ−1−ρk_3(\rho)=e^\rho-1-\rho. With the numbers above, k3(0.4)≈0.0918k_3(0.4)\approx0.0918, while k3(0.1)+k3(0.3)≈0.0550k_3(0.1)+k_3(0.3)\approx0.0550, and the two are not equal.

If we keep the signs of the differences and then take expectations, we can also relate the two differences to KL. Fix the same prefix, and write the distribution that corresponds to the prefill score as πprefill\pi_{\mathrm{prefill}}. Assume that the probability recorded by decode really describes the actual sampling distribution πgenerator\pi_{\mathrm{generator}}, and that the three distributions can produce the same set of tokens and all give these tokens nonzero probability. Under these conditions, if we negate the differences above and take the expectation over decode samples, we get

KL(πgenerator∥πθ)=KL(πgenerator∥πprefill)+Ey∼πgenerator[log⁡πprefill(y)−log⁡πθ(y)].\begin{aligned} \mathrm{KL}(\pi_{\mathrm{generator}}\|\pi_\theta) ={}&\mathrm{KL}(\pi_{\mathrm{generator}}\|\pi_{\mathrm{prefill}})\\ &+\mathbb E_{y\sim\pi_{\mathrm{generator}}} \left[\log\pi_{\mathrm{prefill}}(y)-\log\pi_\theta(y)\right]. \end{aligned}(13)

This formula omits the prefix condition that the three distributions share. The first term is indeed the KL between decode and prefill in the generator. For samples drawn by decode, we can estimate this term by subtracting the prefill log-prob from the decode log-prob and averaging the differences. The second term, however, also uses decode samples, so it can be negative, and it usually does not equal KL(πprefill∥πθ)\mathrm{KL}(\pi_{\mathrm{prefill}}\|\pi_\theta). So we can use this decomposition of k1k_1 to quantify the difference inside the generation side, but we also need to keep the sign of the second term. These two quantities are not two nonnegative KL terms, and this decomposition alone does not let us attribute the error to two causes that do not affect each other.

If the decode prefix is already corrupted, the per-token log-prob difference can still be split algebraically into the two parts above, but we can no longer apply the KL interpretation on a shared prefix. In this case, the difference between prefill and decode may contain both a difference in numerical paths and an error in the actual execution state. To separate the two, we can rebuild the cache from the correct prefix, and then force decode to step through the recorded tokens one at a time, as a control path that does not go through a pause and a resumption. If this path also shows a difference, we continue by checking the decode operators and the cache precision. If the difference appears only at request resumption, we check the counts and the state before and after the resumption. The size and the pattern of these differences can help us plan experiments, but the specific cause still has to be confirmed by the results of controlled comparisons and by state records.

After this set of comparisons, we can narrow the investigation to a specific path, but we have not yet found the final cause. The next section follows the branch in which "the trainer and prefill agree, but the actual decode is abnormal", and analyzes an error in restoring the prefix when a request resumes. How to compare tensors inside layers, how to validate weight synchronization, and how to decide whether a fix is complete are discussed later, in the complete debugging workflow.

5. From a KL Spike to request resumption: debugging a prefix misalignment

TL;DR: If tokens whose results are still in flight are handled incorrectly when generation resumes, the recorded prefix can differ from the prefix the engine actually uses, so the log-prob difference no longer has the same-prefix KL interpretation. Validating a fix requires reproducing the relevant pause and resumption conditions and checking the state before and after the transfer, since the absence of a KL Spike alone is not enough.

Suppose a KL Spike appears in monitoring. We export the abnormal rollout and compare it with the scheduling records of the generator. We find that one of the responses was temporarily removed from the generation batch by the scheduler before it finished. Later, the response rejoined the batch and continued to generate the rest of its tokens. What is resumed here is the same unfinished generation request. The engine needs to continue from the existing tokens, and it does not start a new response from the prompt.

When we compare the log-prob of each token, we find that the large differences are concentrated in the part of the response that was generated after this request rejoined the batch. The generator gives these tokens high probabilities, while the trainer, which recomputes on the complete recorded prefix, considers them almost impossible. This gives us a concrete place to check: the token records and the execution state before and after the pause. However, the fact that the request resumption and the anomaly appear at nearby positions does not directly prove the cause. Below we use a simplified set of tokens to show one kind of error that could cause this behavior, and then look at how to validate it with controlled comparisons.

To reduce the time that the GPU spends waiting for the CPU, an asynchronous scheduler running on the CPU may schedule later computations ahead of time. So even after the scheduler has decided to stop scheduling new computation for a request for now, work that was submitted to the GPU earlier may not have reported all of its results yet. At this point, the GPU may have already generated some tokens, while the CPU has not yet finished processing the results returned for these tokens. So "how many outputs have been reported" and "how far generation has progressed, counting the results that are still on their way back" describe generation up to two different points. When the scheduler is ready to let this request continue, it sends the required tokens and state information to the worker that runs the GPU computation. In this article, we call this information the resume message. If the count and the token list in the message describe generation up to different points, the prefix that the model uses next can be wrong.

We continue with the token sequence from earlier. Suppose the prompt is [P1, P2, P3], the reported outputs are [A, B], and another result, C, has already been generated but has not returned yet. At this point, the request is skipped for now, and then it becomes ready to resume. Suppose the resume message gives the worker the output count of 3, which includes C, together with the token list that does not yet include C. We then get the following inconsistent state.

Reported token list
P1P2P3AB
Not returned yet
CCounted in the output count of 3, but missing from the list above
Outputs restored from the last 3 items
P3ABP3 is mistaken for an output
Figure 2: In this hypothetical resume message, the output count and the contents of the list describe generation up to different points. Every token ID in the list is valid, so an ordinary range check cannot find this error.

If the worker restores the outputs with the rule "take the last 3 items of the list", it gets [P3, A, B]. The correct outputs, however, should be restored as [A, B, C] after C has returned. After this misalignment, the positions, the context, or the cache state that are rebuilt afterwards may also shift. The program can still keep running, because P3 is a valid token, and the generator can still compute a valid probability distribution from its current state.

At this point, a high probability from the generator only shows that this token is reasonable in the context that the generator actually used. It does not show that the recorded response is also reasonable. When the trainer recomputes probabilities from the normal record, it is already using a different prefix, so the two sides may show large differences continuously from this point on. If the wrong state further leads generation into a repeated passage, the later anomalies become more and more visible. The most important place to check, however, is still the first divergence around the request resumption.

Next, we can use the prefill recomputation from the previous section. We give the correct recorded sequence to the generator, rebuild the cache, and compute the log-probs of the same tokens. If this score is close to the trainer but far from the saved decode result, we can follow Figure 1 to check the incremental computation and the execution state. We can then use a decode that is forced to step through the recorded sequence to separate these two sources. Here, the agreement between the prefill score and the trainer is evidence that helps us narrow the investigation. If we only resubmit the same prompt and let the model generate a response again, however, the scheduling timing may have changed, and the original trigger condition may be missed. So these two kinds of "replay" provide different evidence.

To validate this hypothesis, we need to construct the situation in which "the request still has results on their way back, but is about to resume", and then check the count and the list contents in the resume message. For the state machine above, one fix is to allow the request to resume only after all in-flight results have been reported. Another design passes the unfinished state explicitly, and lets the receiver handle it with the same rules. Whichever approach we use, the count, the token list, and the state in the resume message must all describe the same point in generation.

When validating the fix, besides confirming that the original wrong state is rejected, we also need to confirm that the test actually reached this resumption path. Seeing the KL Spike disappear is not enough, because it cannot tell us whether "the error has been fixed" or "this run did not hit the trigger condition". If a production system keeps running for now by dropping abnormal rollouts, it should also record the number of dropped rollouts and the characteristics of these samples. For example, long responses are more likely to go through a pause and a resumption. If these responses are filtered out more often, the length distribution of the training data may also change.

In this example, we started the investigation from the abnormal tokens kept by monitoring. We then narrowed the scope by comparing prefill with decode, and finally used the state records before and after the resumption to explain how the error happened. After finding the cause, we still need to validate the fix with a specially constructed resumption scenario. If the original zero-difference test did not cover this state transition, we should add this scenario to it, so that the problem is not missed again in the future.

6. If the two sides agree on log-probs, why is sampling still not guaranteed to be correct?

TL;DR: Even if the two sides report identical log-probs, those probabilities may not describe the distribution from which tokens were actually sampled. For speculative decoding, candidate-block scores, acceptance and resampling rules, and cache state after rejection each need separate validation.

The tests so far check whether the same token gets the same score on both sides. But if we want to use the probability returned by the generator as the denominator of the importance ratio, we also need to confirm that the output token was really sampled from this distribution. For example, a temperature transform can make the actual sampling distribution differ from the probability that the interface returns. In a more complex candidate verification process, every scoring step can be completely correct, and an error in the acceptance rule can still make the final samples follow the wrong distribution.

To separate these two things, we write the log-prob returned by the interface as ℓtreport\ell_t^{\mathrm{report}}, and the difference computed from the logs as

ρ^t=log⁡πθ(yt∣ht)−ℓtreport.\widehat\rho_t=\log\pi_\theta(y_t\mid h_t)-\ell_t^{\mathrm{report}}.(14)

The ρt\rho_t defined earlier uses the actual sampling distribution πgenerator\pi_{\mathrm{generator}}. So ρ^t\widehat\rho_t equals ρt\rho_t only when the value returned by the interface really equals the log-prob under this distribution, and only then can we apply the earlier KL interpretation to the logged values. For tokens produced by decode, the ℓtdecode\ell_t^{\mathrm{decode}} that we recorded when we split the computation paths is exactly the ℓtreport\ell_t^{\mathrm{report}} here. So the per-token algebraic split always applies to the logged differences. To interpret such a difference as the true log-ratio, however, we also need to confirm that the returned value describes the actual sampling distribution. The two examples below show how the probability return mode and the sampling implementation can each break this precondition.

6.1 Temperature changes the sampling distribution, while the raw log-prob may stay the same

So far we have set the temperature to 1, so that the raw softmax distribution is the same as the actual sampling distribution. Once the production configuration changes, however, we need to state clearly which distribution the training objective uses. For example, if we want to train the temperature-transformed policy itself, the trainer should use the same temperature when it computes probabilities. Then both sides compare transformed probabilities, and the numerator and the denominator have consistent definitions.

Another setting keeps the distribution of the raw model as the training target. It generates samples from the temperature-transformed distribution, and then corrects the expectation with an importance correction. In this case, the probability described by the raw log-prob and the actual sampling probability are two different quantities. We write the logits output by the generator as zvz_v, the raw distribution as πraw\pi_{\mathrm{raw}}, and the behavior policy that actually produces the samples as πbehavior\pi_{\mathrm{behavior}}:

πraw(v)=ezv∑uezu,πbehavior(v)=ezv/T∑uezu/T,T>0.\pi_{\mathrm{raw}}(v)=\frac{e^{z_v}}{\sum_u e^{z_u}},\qquad \pi_{\mathrm{behavior}}(v)=\frac{e^{z_v/T}}{\sum_u e^{z_u/T}},\qquad T>0.(15)

Suppose the vocabulary contains only A and B, and the logits are [0,log⁡3][0,\log3]. The raw probabilities are then [0.25,0.75][0.25,0.75]. When T=2T=2, the actual sampling probabilities become about [0.366025,0.633975][0.366025,0.633975]. If the target distribution of the trainer is still the raw distribution, and both sides return raw log-probs, the token-by-token difference is zero. However, to correct the single-step action expectation under the behavior distribution to the target distribution of the trainer, the correct weights should be

0.250.366025…≈0.683013,0.750.633975…≈1.183013.\frac{0.25}{0.366025\ldots}\approx0.683013,\qquad \frac{0.75}{0.633975\ldots}\approx1.183013.(16)

If we mistakenly use the raw probability as the denominator, both weights become 1. We can see the effect directly. A appears in the actual samples with a proportion of about 0.366025. Multiplied by the correct weight, this proportion returns to 0.25, the probability of A under the target distribution. If the weight is 1, the proportion stays at 0.366025. In other words, although the numerical computations on the two sides agree exactly, the expectation has not been corrected to the target distribution that we want to optimize.

So when integrating a sampling interface, we need to confirm whether the returned log-prob describes the probability before or after the temperature transform. For example, the vLLM documentation for --logprobs-mode distinguishes raw_logprobs from processed_logprobs. The default is raw_logprobs, while processed_logprobs describes the generation probability after processing such as temperature, top-k, and top-p. The documentation also notes that prompt log-probs do not go through this sampling processing, so the two modes return the same results on the prompt. When we rescore with prefill, this is an interface difference that we need to check.

In addition, top-k, top-p, and constrained decoding set the probability of some tokens to zero. If the target policy still gives these tokens nonzero probability, the samples that are actually drawn cannot recover the full target action expectation without bias. The reason is that the excluded tokens never appear in the samples, so they never get a chance to contribute their share of the expectation under the target distribution.

So before debugging starts, we should first confirm the definition of the probability. Otherwise, we may spend a long time aligning two numbers without ever confirming that they are the probabilities that the training objective really needs. Even when the definition used by the interface is correct, however, the sampling process itself can still go wrong.

6.2 Candidate verification can use the wrong probability while the scores on both sides still match exactly

In speculative decoding, a cheaper drafter usually first proposes several candidate tokens. Then the target model, the verifier, scores these candidates and decides which tokens to accept and at which position to resample. Multi-Token Prediction, or MTP, provides one way to produce candidates. MTP itself is a training objective for predicting multiple tokens, or the additional prediction module that goes with this objective, so it is a different concept from speculative decoding. For example, the DeepSeek-V3 technical report explains that at inference time, one can either discard the MTP modules and use only the main model, or let the MTP modules take part in generating candidates.

For RL, what we care about is whether the final output, after the candidates have been verified, still follows the distribution that the main model is supposed to sample from. Take the standard rejection sampling algorithm from the speculative sampling paper as an example. We fix the prefix hh, write the target distribution of the verifier as πverifier\pi_{\mathrm{verifier}}, and write the proposal distribution that the drafter actually uses as πdraft\pi_{\mathrm{draft}}. Both distributions already include their own temperature transforms, truncation, and other processing. Below we omit the shared prefix condition hh. For a token yy proposed by the drafter, the probability that it is accepted is

a(y)=min⁡ ⁣(1,πverifier(y)πdraft(y)).a(y)=\min\!\left(1,\frac{\pi_{\mathrm{verifier}}(y)}{\pi_{\mathrm{draft}}(y)}\right).(17)

So the probability of getting yy through the acceptance branch is πdraft(y)a(y)=min⁡(πverifier(y),πdraft(y))\pi_{\mathrm{draft}}(y)a(y)=\min(\pi_{\mathrm{verifier}}(y),\pi_{\mathrm{draft}}(y)). If the candidate is rejected, we normalize max⁡(0,πverifier−πdraft)\max(0,\pi_{\mathrm{verifier}}-\pi_{\mathrm{draft}}) and sample one token from that distribution. The total rejection probability is exactly the sum of these positive differences, so the rejection branch supplies exactly the part of the verifier probability that the acceptance branch has not covered. With the two branches combined, the final behavior distribution equals πverifier\pi_{\mathrm{verifier}}. This conclusion, however, requires one condition. The proposal probability used by the acceptance rule must be the same as the probability that was actually used to generate the candidate.

For a block of candidates proposed in sequence, verification starts at the first position. If the first rejection happens at some position, a new token is sampled at that position, and all later candidates are discarded. If the whole block of candidates is accepted, one more token is sampled from the verifier distribution after all the candidates. This token is usually called the bonus token. It comes directly from the verifier, so it does not need to pass an acceptance test against the draft.

So when the conditions above hold, what should be passed to RL is the log-prob of the final token under the verifier distribution. The same definition applies whether the token is an accepted candidate, a token resampled after a rejection, or a bonus token. For a resampled token, we especially need to distinguish two quantities. RL needs the behavior probability that is obtained by combining the acceptance branch and the rejection branch. The residual probability used in the rejection branch carries an extra condition, namely that a rejection has already happened. The two must not be mixed up. If the target distribution of the trainer differs from the sampling distribution of the verifier, we still compute the importance ratio with the earlier definition, and there is no need to multiply by an extra correction for the draft probability.

If the acceptance rule uses the wrong draft probability, the distribution guarantee above no longer holds, but a token-by-token comparison of log-probs may not find this error. We can construct an example on a fixed prefix. The verifier gives A and B the probabilities [0.6,0.4][0.6,0.4], and the raw probabilities of the drafter happen to be the same. But the drafter uses top-1, so it actually proposes A every time. If the verification program wrongly reads the probability before truncation, which is 0.60.6, the acceptance probability of A is computed as 0.6/0.6=10.6/0.6=1, and the output is A every time. The correct denominator is the actual proposal probability, which is 11. Only then is A accepted with probability 0.60.6, and the remaining rejection branch produces B.

In this incorrect implementation, the generator can still return the verifier value log⁡0.6\log0.6 for A, and the trainer also gets log⁡0.6\log0.6 when it recomputes the log-prob of A, so the logged ρ^=0\widehat\rho=0. But the actual behavior probability of A has become 11, and the true log-ratio should be ρ=log⁡0.6−log⁡1≈−0.511\rho=\log0.6-\log1\approx-0.511. This case is similar in one way to the earlier example of the old log-prob recomputed by the trainer. Even when the logged ratio looks equal to 1, we cannot conclude that the samples come from the trainer distribution. If PPO still uses the probability returned by the interface as the denominator, this ratio cannot correct the oversampling of A. Furthermore, even if we knew the true behavior probability, B never appears in the samples, so importance weighting cannot restore the contribution of B to the target expectation. What needs to be fixed here is the acceptance rule itself.

We fix the same prefix, and simulate the correct and the incorrect sampling processes for this two-token example 100,000 times each. The frequencies of A are 59.808%59.808\% and 100%100\%, respectively. The correct implementation does not give exactly 60%60\% either. For independent samples, the standard error of the frequency is 0.6×0.4/100000≈0.00155\sqrt{0.6\times0.4/100000}\approx0.00155, so the deviation observed in this run is consistent with normal sampling variation. This example shows that a frequency check can find errors that a token-by-token score check misses. When judging the result, however, we still need to take the number of samples into account, and we should not treat every deviation as a bug.

6.3 The numerical path during verification and the cache after a rejection both need separate checks

Even when the acceptance algorithm itself is correct, speculative decoding adds one more numerical path that needs to be aligned. Ordinary decode usually processes only one new position for each request, while the verifier processes several candidate positions at once, so the matrix shapes, the attention kernels, and the reduction order may all change. The mathematical proof of distribution preservation above assumes that the same prefix corresponds to the same target distribution. If floating-point computation changes this target distribution, that change needs a separate check, and the mathematical proof cannot replace this check. The vLLM documentation also describes theoretical distribution preservation separately from the stability of the actual log-probs.

In addition, candidate verification adds state handling after a rejection. Suppose the drafter proposes A, B, and C, and the verifier accepts A, rejects B, and resamples D at the position of B. The committed output should then be A, D. At this point, the temporary cache for B and C must be rolled back or masked, and the state for D must also be built correctly before later computation uses it.

Proposed candidates
ABC
Verification result
A: acceptedB: rejectedC: discarded
Commit the output and prepare the next round
AD: resampledThe valid prefix for later computation is the original prefix followed by A and D. Later computation must not read the state of B or C.
Figure 3: Even when all candidate scores are correct, if the state left by the candidate branch is still used after a rejection, the next round of computation departs from the committed sequence. To find this kind of error, we also need to check the first computation after the state has been rolled back.

This connects to the earlier request resumption case. We need to confirm separately whether the scoring paths agree, whether the acceptance rule preserves the target distribution, and whether the later state corresponds to the committed sequence. If we only turn off speculation and see the anomaly disappear, we still cannot tell whether the problem lies in the computation shape during verification, in the acceptance probability, or in the state after a rejection. So in the debugging workflow later in this article, each of these checks needs to stay separate.

7. How should debugging move from an abnormal sample to a regression test?

TL;DR: Debugging should start from an abnormal sample that can be reproduced and use controlled comparisons to locate the first point of divergence. Validation after a fix must cover nonzero parameter updates and weight synchronization, the original failure conditions, and the production configuration, while also confirming that the checks detect deliberately introduced errors.

When we bring the checks above into our own system, we can first confirm what is being compared, then build controlled comparisons, then narrow the scope based on the results, and finally validate whether the fix works. When integrating a model for the first time, we can start from a small set of short samples. When a KL Spike appears in production training, we should first keep the failed rollout and the execution events that triggered the problem. There is another situation worth investigating. With fixed weights, turning speculation on or off may cause a reproducible change in statistics such as token frequencies or response lengths, while the score difference in the logs does not change accordingly. In this case, we need to check the sampling process. These statistical changes alone, however, cannot tell whether the problem comes from the acceptance algorithm, from the numerical path during verification, or from normal sampling variation.

7.1 First save inputs that can reproduce the problem, and confirm that both sides compare the same quantity

When debugging starts, the first thing we need is a sample record that can be rescored. If all we have is a curve of means, or only the decoded responses, we cannot complete the comparisons below. The following table lists the information to save:

What to save and checkAmbiguity to rule out
The complete token IDs, the prompt boundary, the valid generated positions, and the position IDs, mask, and multimodal inputs that were actually usedWhether the two sides predict the same token on the same prefix, and whether padding was included or EOS was left out
The ℓtreport\ell_t^{\mathrm{report}} returned by the generator during generation, the log-prob from the actual loss path of the trainer, and the dtype of eachWhether old log-probs recomputed by the trainer replaced the generator records, and whether precision was lost on export
The weight snapshot identifier, the synchronization records, and the tensor names, shapes, dtypes, and content digests that can be checkedWhether the same version number really means the same weights, and whether an adapter, a router, or a group of experts was left out of the transfer
The temperature, top-k, top-p, logits processors, and the log-prob return modeWhether the comparison uses the raw distribution or the processed distribution that was actually sampled from
The batch, parallel, and cache configuration, and the points where requests were paused and resumed and where weights were switchedWhich execution conditions need to be kept for reproduction
When candidate verification is enabled: the versions of both models, the candidates and acceptance results, the resampled tokens, the actual proposal probabilities, the verifier probability of each candidate, and the valid cache length before and after rollbackWhether the acceptance rule used the actual proposal probability, whether rejected candidates also kept their scores, and whether the state after rollback corresponds to the committed sequence

When checking the weights, we first need to convert the sharded or packed representations back to the same logical tensors. If the two sides use different storage layouts, we cannot compare their byte digests directly. In addition, if the system allows the weight version to change during generation, we need to map each token or each generated segment to its specific version, and then compare only the parts with the same version. Otherwise, the effect of parameter version lag mixes into the execution difference that we actually want to check.

If the compared objects are not aligned yet, we should first fix the inputs, the indexing, or the probability definition. If the log-probs that the generator returned during generation are missing, we need to add recording and collect the data again. Probabilities that the trainer recomputes afterwards cannot replace the results that the generator actually gave at the time.

If we are investigating a production failure, we should start from the saved original samples and the weights at that time, and first complete the path comparisons below. A normal response that is regenerated under a new configuration may not have gone through the original trigger condition, so it cannot replace the failed response. We change the configuration in a targeted way, or add new comparisons, only when we need to isolate a specific factor.

If we are integrating a model for the first time, or preparing to build a regression test, we can start from short samples. With temperature 1, no truncation, dropout turned off, and a clean cache, we collect the generation results and the corresponding scores. If we can make the two sides use matching numerical paths, we build a strict zero-difference reference. If that is not possible yet, we can still first check the inputs and the state, and then measure the numerical deviation between each pair of paths. Later parts of this chapter discuss how to use these nonzero results for debugging.

If the system uses MTP or another drafter to speed up generation, we can first turn off candidate generation to build a reference with ordinary token-by-token sampling, and then turn on the candidate verification path for comparison. When investigating a failure that has already happened, however, we need to keep the original candidates and acceptance records, because generating a new set of candidates may avoid the original trigger condition. Also, the acceptance rule needs to use the actual proposal probabilities from the time when this set of candidates was generated. Probabilities that are recomputed after the drafter has been updated cannot replace them.

The probabilities saved in the original samples also determine how we should compare. If raw log-probs were saved, we can compare them with the raw log-probs recomputed on both sides. If only processed probabilities were saved, we first need to replay the same temperature transform, truncation, and other processing on the logits of the trainer and of prefill, and then compare the probabilities of the selected tokens. To do this, we must be able to recover the complete state that the processing needs. The raw log-prob of a selected token alone is usually not enough to recompute the normalization term after the transform. If this state can no longer be recovered, we can first use the original samples for a raw score comparison between the trainer and prefill, and collect a separate numerical comparison for decode. This still does not fill in the missing decode probability records of the original failure, however, so we need to continue the investigation by adding recording or by triggering the original condition again.

7.2 First check whether the generator can reproduce its own scores and state

After saving the scores from generation time, we can first run several comparisons on the same sequence inside the generator. This way, we can start checking the generation side even before the trainer is connected. All of these comparisons need to fix the token IDs, the weights, the prediction positions, and the probability definition. Also, for each recomputation, we need to state clearly which states are rebuilt and which states are reused from existing results.

In forced decode, the key is to split "computing the distribution of the next token" and "choosing the next input token" into two steps. At each step, we first read the raw logits or raw log-probs of the current model and record the score of the token to be compared. Then we feed the next token of the saved sequence as input and advance the cache. This keeps the incremental computation, and also avoids the changes to the sequence that resampling would cause. If the framework forces a choice by masking the other tokens, the score must be taken before the masking. Otherwise only one token remains in the processed distribution, and its log-prob becomes zero. If the framework does not expose these interfaces yet, we can first run the repeated prefill and batch comparisons. To complete the decode comparison, however, we still need to add recording on the engine side.

What to compareWhich condition changesHow the result guides the next step
Repeated prefill on the same sequenceClear the old state each time, and keep the execution configuration and the batch the sameIf there is still a difference, first investigate the repeatability of this path. If the results are equal, continue with the next two comparisons. Equal results here do not directly imply that decode also agrees.
Within one stage, running alone versus running together with other requestsCompare prefill and forced decode separately, and record the batch at each step. Test prefill chunking in a separate comparisonLocate the divergence layer by layer between the two batch configurations of this stage. For MoE, also compare the router scores and the selected experts
Prefill versus decode that steps through the fixed sequenceFirst let both process this sequence alone, fix the parallel configuration, and exclude interference from other requestsIf there is still a difference, compare the actual operators and states of the two stages, and keep the input and the shapes at the first divergence
Continuing with the existing state versus rebuilding the state after clearing itCompare the two sources of state at the same prefix boundary, then force both to step through the same tokensCheck whether rebuilding changes later scores, and keep the boundary state and the position of the first divergence

The first two rows of the table check repeatability under a fixed configuration and batch invariance, respectively. Only the third row starts checking whether prefill and decode agree. An implementation can be stable on every run and still produce two different sets of results. So passing the earlier checks cannot replace the later ones. A single-request comparison does exclude interference from other requests, but the multi-row computation of prefill and the single-row computation of decode still have different shapes. So what the third row measures is the overall difference caused by switching stages, and this difference cannot be attributed directly to attention or to the cache. If the difference appears only after other requests are added, we should first locate it between the two batch configurations within the same stage, and then redo the cross-stage comparison.

The last row of the table is especially useful in multi-turn conversations. Suppose the first turn has generated A, B, and C, and the token sequence returned by a tool is R. The second turn then needs to continue from the prefix (x,A,B,C,R)(x,A,B,C,R). One path can keep the cache left by the step-by-step generation of the first turn, and then process any trailing generated tokens that have not been written into the state yet, followed by the tool-result tokens R. The other path clears the state and runs prefill again on the same complete prefix. After both paths reach the same boundary, we fix the batch, the kernels, and the input tokens of the later decode, and compare the scores of the second turn. Even if the token IDs, the positions, and the weight version are all correct, the numerical states built by these two approaches can still differ. So this observation alone cannot prove that the cache is stale or corrupted.

To further separate state computation from state management, we can start from the same incremental state that has already been checked. One path continues computing directly. The other path goes through the save, restore, or reuse process that we want to investigate, and then continues with the same decode configuration. If the result is normal when the computation continues directly, but a new difference appears after the state handoff, we need to check the logical state, the valid length, and the position mapping before and after the handoff, one item at a time. For tensors that are supposed to be passed through unchanged, we also need to confirm that their values have not changed. If, in a comparison without any state handoff, the clean token-by-token rebuild still differs from prefill, we can first locate the problem along these two ways of building the state. How large the difference is, and whether it concentrates at page boundaries, only helps us choose where to check. Neither observation can by itself prove which kind of cause the problem belongs to.

For models with recurrent state, we also need to check the state tensors at the boundary. Full or chunked prefill and token-by-token updates can be equivalent in exact arithmetic. But because the computations are grouped differently, they can leave different numerical states, and later tokens keep reading these states. A public alignment analysis of Gated DeltaNet gives an example of this kind of problem. So besides confirming that the score of the current token agrees, we also need to look at what state this computation writes, and whether the next computation reads the correct state.

There is another comparison in multi-turn records that is easy to confuse. When A, B, and C are put into the prompt of the second turn, the prompt log-probs give the prefill rescoring of the first-turn response. These results can be used for the diagnosis above, but they must not overwrite the decode log-probs that were saved when the first turn was generated. The behavior probability that RL uses must keep its actual source from generation time. The same text becomes part of the prompt in the next turn, but this does not allow us to treat the recomputed probability as the probability with which the text was originally sampled.

7.3 Compare the paths that actually ran, then choose the next experiment

After checking the generator itself, we take three scores for the same weights and the same sequence: the trainer score, the generator prefill score with a clean cache, and the score saved during generation. The comparisons below mainly concern tokens produced by decode. The first token of a response is usually sampled from the prefill output of the prompt, so it needs to be marked separately. Before computing the differences, we also need to confirm that the number of valid tokens is as expected, and that there are no missing or non-finite values. If abnormal tokens are filtered out in advance, or if an empty result is allowed to pass, the later comparisons lose their meaning.

An "anomaly" in the table below means a difference outside the known numerical range of the corresponding path. With a strict zero-difference configuration, any nonzero difference needs investigation. If the two sides have a numerical deviation that has already been measured, we need to build a separate reference for each of the two comparisons. The later section on differences that cannot be brought to zero yet describes the method. Even if the result this time falls within the existing range, this only shows that no new anomaly was observed. It does not prove that the path has no error.

ObservationWhat to do nextEvidence to keep
Trainer versus prefill is abnormal, and prefill versus decode matches the baselineCheck the weight representation, the precision, and the parallel configuration on both sides, then compare the two full-sequence forward computations layer by layerThe input, parameters, and output of the first layer that diverges
Trainer versus prefill matches the baseline, and prefill versus decode is abnormalRebuild the cache from the correct prefix and force decode to step through the recorded tokens. If continuous stepping is also abnormal, check the incremental operators and the cache precision. If the anomaly appears only at a pause and resumption, check how the state is passed.The position of the first divergence, and the corresponding cache write, cache read, or resumption events
Both comparisons are abnormalKeep both differences and narrow down each of them separately. After fixing one part, rerun the comparison for the other partBoth differences, recorded separately while one factor at a time is changed, so that cancellation cannot hide either difference
The controlled configuration agrees, and the original configuration shows a differenceRestore the original batch, kernels, parallelism, cache, and scheduling conditions one at a time, and check their combinations when neededThe configuration change and the input at which the difference first reappears. Reaching zero alone does not show that the original configuration is free of errors
Candidate block scores are abnormal, and ordinary token-by-token scores match the baselineOn the same prefix, change the length of the candidate block and compare the verifier inputs, masks, and numerical pathsThe candidate position of the first divergence, the block length, and the corresponding tensors inside the layers
Scores before the rejection match the baseline, and the first divergence appears in the computation right after a rejectionConstruct a rejection from the same state, check the rollback and the commit, and then compare with a cache that is rebuilt from the final sequenceThe rejection position, the valid cache length before and after rollback, and the position of the first divergence in the next round
Recomputing the original sample agrees with the saved decode, but the training log shows a differenceCheck the token mapping and the weight version in the logs, then restore the batch composition, sequence concatenation, and mask that the trainer used at the timeThe differences in inputs and scores between training and standalone recomputation, and the index mapping of the logs

When comparing candidate blocks, we also need to align the prefix of each position with the prefixes used by the trainer, prefill, and forced token-by-token decode. If the later candidates have already been discarded after the first rejection, the computations at those positions can only be checked against the original candidate prefix, and their scores must not be matched to the actual outputs after resampling. For the replacement token resampled at the rejection position, we score it with the verifier distribution at that position. In the next round, we then check the real prefix that is formed after this new token is committed. The regression test part later in this chapter explains how to trigger a rejection deterministically in order to check the state rollback branch.

The last row of the table can also help us find prefix errors on the trainer side. For example, suppose two responses are packed into one row and the attention mask does not separate them. Then the second response can see the content of the first. If we take the second response out and rescore it alone, this error disappears. So when reproducing a problem, we need to keep the batch conditions of each side, and we should not assume at the start that the problem can only be in the generator.

After narrowing the scope to one path, we can first compare the embedding output, and then search along the inputs and outputs of each block for the earliest divergence. Only when the inputs and the parameters are aligned can we attribute a new difference to the current operator. If the input that a layer receives is already different, we should keep checking earlier layers. We should not modify this layer first just because its output difference is the largest. For MoE models, we also need to save the router scores and the selected expert indices, so that we can separate numerical changes from discrete routing changes. If the framework supports exporting routes, we can add a comparison on the same input that replays only the expert indices of the generator, and see whether the difference becomes smaller. This comparison tests the effect of the routing choices, but it cannot by itself prove that all other computations are correct.

If the full logits already agree but the log-probs still differ, we can check the final normalization step separately. For a token yy,

log⁡π(y)=zy−log⁡Z,Z=∑v∈Vezv,\log\pi(y)=z_y-\log Z,\qquad Z=\sum_{v\in\mathcal V}e^{z_v},(18)

where zyz_y is the logit of the selected token, V\mathcal V is the valid vocabulary, and log⁡Z\log Z is the log-sum-exp over the whole valid vocabulary. By this formula, the log-prob difference between two paths equals the difference between their zyz_y values minus the difference between their log⁡Z\log Z values. So we should record both quantities, and check which vocabulary range the normalization covers. For example, if the extra columns padded for parallel alignment are not masked, then even when the logits of all valid tokens are the same, the normalizing denominator changes, and this causes an overall shift. If the logits and the vocabulary range are already the same and the only difference is the reduction order, we can consider using a batch-invariant log-softmax to align this step.

After this step, we should be able to narrow the problem down to one specific divergence that can be reproduced. For example, "the same input starts to differ after this operator", or "after this resumption event, the token count and the list contents no longer agree". After fixing this problem, we still need to go back to the original failed input and check whether other differences remain. A system can have several problems at the same time, so removing the first shift does not mean that debugging is finished.

7.4 After the scores match, how do we check the sampling process?

If a sampling algorithm has been added or modified, we should make the distribution check a separate regression test. This test still needs to run even if all the earlier score comparisons have passed. If production shows that "the output statistics changed, but the score difference did not", we can also start the investigation here. However, if the scoring path itself is abnormal, we need to record and locate that anomaly first, so that changes in the verifier distribution do not get mixed into our judgment of the acceptance algorithm.

For standard rejection sampling, we can first check the existing candidate records. We use the verifier probability and the actual proposal probability to recompute the acceptance probability a(y)a(y) that each candidate should have. We then group the candidates by this probability, and compare the actual acceptance rate in each group with the mean of a(y)a(y), while keeping the sample counts and the statistical error. Only candidates that actually went through the acceptance test should be counted here. The candidate suffix that was skipped after the first rejection did not go through another test, so it must not be counted as rejected. In the top-1 counterexample above, the expected acceptance rate is 0.60.6, but the actual rate is 11, and this comparison exposes the deviation directly.

Whether this check works depends on whether we have independently verified where the proposal probability comes from. If the verification module reads the raw probability by mistake, and the check recomputes a(y)a(y) from the same field, the two sides still agree, and the error is kept on both sides. So we need to use the sampling step that actually generated the candidates, together with the transform records, to confirm what the proposal probability really is. For example, the probability of a deterministic top-1 proposal should be 11. Even when the acceptance rate matches the expectation, we still need to check whether the rejection branch used the correct residual distribution. When necessary, we can export the full target distribution and proposal distribution at that position for a small number of diagnostic samples.

Next, we can sample repeatedly on a fixed prefix and compare the actual output frequencies. We can first validate the algorithm with a known distribution on a small vocabulary, and then check the integrated result on a fixed prefix of the real model, while recording the corresponding verifier distribution. If a candidate block contains several positions, the small-vocabulary test should also compare the joint frequencies of two or three consecutive tokens with the product of the per-position conditional probabilities of the target model. This avoids checking only the first position and missing problems in the later conditional distributions. Of course, a finite number of samples may not have enough power to detect a very small deviation, so we also need to record the number of samples and the uncertainty range of the frequencies.

If both the acceptance rate and the output frequencies match expectations, but the outputs still differ between speculation on and speculation off, we can go back to the scoring path of the candidate block, and check whether the change in computation shape has changed the verifier distribution. We also need to remember that random sampling naturally produces different responses. Even with the same seed, the two execution modes are not guaranteed to generate the same result token by token. So a single different response is not enough to conclude that the distribution is wrong.

7.5 When the difference cannot be brought to zero yet, how can we use the measured numerical deviation to keep debugging?

Even when the numerical difference cannot be brought to zero yet, some checks can still be done exactly. For example, the token IDs, the prediction positions, the valid counts, and the completeness of the records should not be relaxed because a numerical tolerance has been introduced. The weights also need to be verified according to the expected sharding, conversion, and quantization rules, and a missing weight must not be explained away as "a small error". The definition of the probability and the mapping between versions also need to be clear. These conditions can be checked even across different kernels, and they do not depend on the numerical results of the whole model agreeing exactly.

For the remaining numerical deviation, we need to measure each pair of paths separately. We can start with the repeated computation and batch comparisons inside the generator, and then extend the samples to the lengths and types that the real data will use. With fixed weights, a clean state, and checked inputs, we change the batch size, the ordering, and the parallel configuration one at a time. We record the distribution of the differences, the maximum, and how the differences change with token position and configuration. This reference, however, only describes the deviation that has been observed. It cannot prove that the deviation is harmless, and it should not be summarized by a single mean. If the GPU architecture, the kernel version, or the quantization path changes, we need to measure again with the same samples and the same statistical definitions.

With this reference, we can keep comparing under the same inputs and configuration. We need to go back to the path branches above and investigate if the difference rises across the board after a synchronization, if a persistent shift that did not exist before appears after some position, or if a certain kind of input starts to show new extreme values. We also cannot require all normal numerical deviations to be independent of position or batch, because different computation paths can naturally produce such patterns. What we need to check is whether these patterns can be reproduced in controlled comparisons, and whether each change has a source that can be checked further.

When accepting results with a tolerance, we also need to state how large an error this check can detect. For example, if we deliberately leave out one weight in a test and the added shift still passes under the existing tolerance, the test has not shown that it can detect this kind of omission. In that case, we can add inputs that expose changes in this weight, reduce the numerical deviation of the related path, or check the synchronized tensors directly. On the other hand, if the usual numerical deviation already causes clipping to remove the PPO gradient contributions of many tokens before the first update, we should first adjust the precision or the computation path that contributes the most, or explicitly choose a corresponding algorithmic correction. Even if such a result is recorded as the baseline, its effect on model updates still exists.

7.6 A fix must be rechecked with real updates, triggered failures, and the production configuration

Passing the comparison with the initial weights only shows that this static forward computation meets expectations. We also need to make the trainer parameters actually change, synchronize them to the generator through the normal path, handle the existing cache, and then generate responses again and compute their log-probs. If the learning rate stays zero, the two sides may always agree even if the synchronization code does nothing at all. For example, suppose the mapping of weight names misses a group of experts. At startup, both sides load the complete weights from the checkpoint, so the first test may look normal. Only after a parameter update, when the new weights of this group of experts are not transferred, does the second test have a chance to expose the problem.

So the minimal validation job should contain two required stages. The job first generates responses with the initial weights and rescores them. Then it repeats this generation and scoring after a nonzero update and a completed synchronization. When checking, besides looking at version numbers, we need to confirm that the trainer parameters really changed and that these changes reached the generator as expected. Both stages must compare the predefined valid tokens and confirm that all values are finite. With a matched zero-difference configuration, every difference should be zero. If the job exits early, has no valid samples, or skips the post-update stage, it should count as a failure.

Besides validating the normal case, we also need to confirm that the check itself can really find errors. For example, in an isolated test, we can deliberately leave out a weight tensor that has changed and that the test samples use, or delete one log-prob, and then confirm that the test rejects these results. For the request resumption problem, we can hold back a generation result that has already finished, so that the resumption logic really meets the state in which "the result has not been reported yet". We then check whether the logic waits as designed or rejects the inconsistent message. After that, we release the result and confirm that the request can continue normally. This way, the test record proves that the trigger condition really happened. It also proves that after the fix, the count, the list, and the cache state satisfy the constraints. Seeing this one KL Spike disappear is not enough to accept the fix.

For candidate verification, we also need to cover several cases separately: all candidates are accepted and a bonus token is produced, a rejection happens at the first position, and a rejection happens at a position in the middle. In each case, we need to check the final committed tokens, the corresponding verifier log-probs, the valid cache length, and the first score in the next round. In a small-vocabulary test, we can make the drafter always propose a token with zero target probability at a chosen position, which guarantees a rejection. In an isolated test on a real model, we can also control the random numbers used for the candidates and the acceptance tests, to make sure that the program enters the rejection branch, and then check the state after rollback. Controlling the random numbers in this way is only for testing state handling, however. To test the distribution, we still need normal random sampling.

After a nonzero update and synchronization, we also need to run these branches again. We confirm that the verifier uses the corresponding target weights, while the acceptance rule still uses the proposal probabilities that were recorded when the candidates were generated. The score comparisons, the sampling frequency checks, and the state rollback tests should also record their results separately. They test different problems, so passing one of them does not mean that the other two have also passed.

Finally, we need to extend the tests step by step to the batch sizes and orderings, sequence lengths, parallel configurations, and input types that are used in practice. Passing a text test does not cover the precision path of image embeddings, and passing a single-GPU test does not cover reductions across GPUs. After the production optimizations are restored, if the two sides use different numerical paths by design, acceptance should confirm several things separately. The exact checks on inputs, versions, and records should pass, and the original failure should be fixed in runs where the trigger condition really happened. The residual differences before and after updates, and under different batches, should fall within the measured range, and the effect of these ratio shifts on updates should have been evaluated. In particular, using the sign of the advantage, we should record how many tokens have their PPO gradient contributions removed early by clipping, and then state clearly what range the training job can accept. Once these conditions are met, the result can be recorded as a regression pass for the current configuration. Whether the strict zero-difference test passed needs to be recorded separately.

In practice, these checks can also run separately. Operator-level or layer-level comparisons on fixed samples can run whenever the code changes. A short job that includes a nonzero update and synchronization can serve as an integration regression test. Pause and resumption, long sequences, and different parallel configurations can go into more complete periodic tests. Production monitoring also needs to keep saving abnormal samples, and to add trigger conditions that were not covered before to the tests. This way, each investigation leaves behind a corresponding check in addition to a fix, which helps us find the same kind of problem in the future.

If we want to turn these checks into a tool, the record format, the aligned token-by-token comparison, the statistics, and the reports can all be reused. What needs to be adapted to a specific framework is mainly how to obtain the prefill and decode scores, how to control the cache and forced stepping, how to export routes, and how to map the synchronized weights. Once the data that these interfaces provide are clearly defined, the diagnostic workflow above can be reused, without requiring every system to use the same training and inference implementation.

8. Debugging Trainer-Generator Mismatch requires checking differences both inside each engine and between the two sides

The debugging workflow above involves several objects of comparison. Inside the trainer, we care whether repeated computation, a change of batch, or packing changes the scores once the weights are fixed. Inside the generator, besides repeated computation, we also need to compare the prefill, decode, and candidate verification paths, and check whether the existing state is still correct when generation continues after a pause. Between the two sides, we check the scores of the same sequence, and whether the weights have been fully synchronized after a parameter update. Each group of comparisons covers a different scope, so even if one check passes, we cannot conclude that the whole system is consistent.

The diagram below is organized by these objects of comparison, and it connects the sources of difference, the monitoring evidence, and the debugging methods discussed throughout the article. Along the internal paths of the trainer and the generator, and along the links between the two sides, you can see where each check sits in the system. The diagram also keeps a separate check for actual sampling, because after the scores agree, we still need to verify that the output follows the expected distribution.

Select a comparison to see its explanation. A blue outline marks the computation paths involved. Double-headed arrows indicate comparisons. Single-headed arrows indicate updates, generation, or data transfer.

TrainerThe scoring path actually used by the training loss
Trainer log-prob
Score frozen before the updateScore under the current parametersRecord changes caused by a parameter update separately from differences under fixed weights.
GeneratorRecomputation, generation, and state maintenance
Prefill full-sequence log-prob
Recorded decode log-prob
Trainer receivesGenerated sequences and probability recordsGenerator generates
Compare differences within each engine and between the engines separately

The trainer and generator may each reproduce their own results while still assigning different probabilities to the same token. The same distinction applies inside the generator. Stable prefill results do not establish that decode, cache handling, or sampling is correct.

If two scoring paths already disagree, start with a controlled comparison of those paths. If the scores agree but the output statistics change, check the sampling distribution. Each comparison rules out only some possible causes, so the results need to be considered together.

Both sides must score the same token after the same prefix

Before comparing probabilities, check the token IDs, prompt boundary, EOS, padding, attention mask, and position IDs. For inputs that include images or other modalities, also check the corresponding input representations. The positions in both records must refer to the same token, with exactly the same preceding prefix. If the correspondence is wrong, correct it before comparing the values.

To investigate computation differences, also fix the weights and confirm that both records contain raw probabilities or probabilities with the same processing applied. Check the number of valid records and look for missing or non-finite values. If the generation records were not saved, collect them again. Recomputing probabilities afterwards cannot establish what decode actually computed during generation.

Trainer scores can change with the batch even before a parameter update

Turn off dropout, fix the parameters and inputs, and repeat the log-prob computation through the path used by the training loss. Then score the same sequence alone and in the training batch, or change only the packing arrangement. If repeated runs disagree, or the scores change with the batch or packing, inspect the mask, numerical precision, parallel reductions, and log-softmax along the trainer computation path.

Probabilities may change as expected after a parameter update. Record that change separately from computation differences with fixed weights. The frozen old log-probs also represent only the earlier trainer scores. They cannot replace the probabilities that the generator used to sample the tokens.

After repeated prefill agrees, compare different batches

Fix the weights and the complete token sequence, and start each prefill run from a clean state. First keep the batch and chunking unchanged to check whether repeated runs produce the same log-probs. Then vary the request composition, ordering, and prefill chunking separately to see whether the scores of the same sequence change.

If changing the batch or chunking introduces a difference, locate the earliest divergence between those prefill runs. Record the kernels actually called and the intermediate precision. For MoE, also keep the router scores, expert indices, and mixing weights. Repeatability under one configuration does not establish agreement under a different batch or after switching to decode.

Locate prefill and decode differences in the operations that actually run

Let prefill and decode process the same sequence separately, with fixed weights and parallel configuration. Force decode to follow the recorded tokens and retain each score before token selection. This excludes interference from other requests, but full-sequence and token-by-token computations still have different shapes. Check the actual backend dispatch, MLA projection order, MoE routing, KV precision, and recurrent state updates.

Once the first differing operator or state is found, compare that part in isolation. With candidate verification, also compare the verifier scores for a candidate block against token-by-token scores on the same prefix. The same backend name in the configuration does not establish that these paths execute the same floating-point operations.

When generation resumes, check that the cache still matches the token record

Start both paths from the same verified incremental state. Let one continue directly and let the other pause and resume, or reuse the cache. Force both paths to use the same subsequent tokens, with the same batch and decode configuration. If only the path with a state handoff diverges, compare the in-flight tokens, valid lengths, positions, page mappings, and cache parameter versions before and after that handoff.

If the state built token by token differs from a prefill reconstruction even without a handoff, compare those two construction processes first. After a rejected candidate, also check that the temporary cache for the discarded suffix was rolled back or masked, the resampled token was committed, and the next computation uses the correct state. The position of an anomaly helps choose a comparison. Controlled comparisons and state records are still needed to confirm the cause.

Matching scores still require a check of the sampling distribution

Fix the prefix and target distribution. Record the temperature, truncation settings, and whether the interface returns raw or processed log-probs. Check which tokens the target policy and the actual sampling policy can each produce. With candidate verification, independently confirm that the proposal probabilities come from the computation that generated the candidates. Then check acceptance rates, residual sampling after rejection, bonus tokens, and the verifier probabilities returned to the trainer.

After these checks, compare output frequencies with the target probabilities and account for uncertainty from the finite sample size. Equal scores can coexist with an incorrect output distribution when the acceptance rule or sampling implementation is wrong. Conversely, one different generated answer does not establish that the distribution is wrong.

Compare trainer and prefill to locate differences between full-sequence implementations

Use the same logical weights, sequence, and probability definition. Let the generator rescore from a clean cache. Keep the original batch configuration of each side and save these configurations with the results. Compare the embedding output and then each layer. At the first differing output, confirm that the operator inputs and parameters agree before inspecting sharding, quantization, LoRA representation, activation and accumulation precision, reduction order, routing, and expert computation.

At the output, also check the valid vocabulary and probability normalization. Even when trainer and prefill agree, compare prefill against the recorded decode results as well. The full-sequence comparison has not yet verified incremental computation or generation state.

The effect on PPO depends on the total difference between trainer scores and generation records

Compare the current trainer log-probs with the decode records saved during generation. To isolate execution differences, select samples for which both sides used the same weight version. First compute the per-token log-prob differences and importance ratios. For a PPO objective whose ratio denominator is the probability recorded during generation, also use the sign of the advantage to count how many tokens have their PPO gradient contributions removed early by clipping.

Adding prefill scores helps identify which part of the computation differs and whether two differences cancel each other. Per-token log-prob differences add across paths, but k3 does not. To interpret a sample mean as a KL estimate, also confirm that the recorded behavior probabilities describe the actual sampling distribution and that every comparison uses the same prefix.

Verify weight synchronization after the parameters actually change

Save the trainer snapshots before and after an update, confirm that the parameters changed, and synchronize through the normal path. Check that the corresponding shards, adapters, and experts received the new weights, and compare the logical tensors on both sides. Clear or update the cache as required by the design so that old parameter-dependent state is not reused.

After synchronization, generate and score new answers, then repeat the comparison between the two sides. In an isolated test, omit a tensor that changed and is used by the test sample to confirm that the check detects the error. An identical version number, or agreement with a zero learning rate, is not sufficient evidence that synchronization succeeded.

Keep abnormal samples so the original failure can be tested again after a fix

Along with valid token counts, means, maximum differences, quantiles, and counts above a threshold, save the corresponding rollouts, weight versions, batch configurations, and scheduling and cache events. Search along the sequence for the first divergence rather than looking only at the largest difference. Empty records, missing or non-finite values, or anomalies filtered out in advance leave the check without complete evidence.

After the fix, confirm that the original trigger occurred again, then check the results after a nonzero update and under the production configuration. Strict zero difference and agreement within a measured tolerance are different outcomes and should be recorded separately. Keep separate results for scoring paths, sampling distributions, and state restoration so that the coverage of the fix is explicit.

Figure 4: Inside each of the two regions, the checks concern the behavior of one engine. Between the two sides, the checks concern score differences and weight synchronization. The same symptom can have several sources, so you still need the corresponding evidence to tell them apart.

9. Closing Remarks

Identical weights do not automatically mean that the trainer and generator use the same policy distribution. When KL or an importance ratio changes, we therefore need to distinguish the contribution of parameter updates from the contribution of the execution process. A zero-difference test provides a controlled way to compare the computations. Probabilities and state saved during generation give us evidence for investigating anomalies in production. For numerical differences that we cannot yet eliminate, we also need to measure the conditions under which they occur and their effects on model updates. With these comparisons in place, we can more confidently distinguish the effects of the algorithm from differences introduced by system execution when discussing distribution shifts in asynchronous training, importance sampling, or KL regularization.

GRPORL infrastructure — reads

Comments

Cite
@misc{zhang2026trainergeneratorconsistency,
  author = {Zhang, Yihua},
  title  = {Everything You Need to Know About Trainer-Generator Mismatch: Sources, Monitoring, Zero-Difference Tests, and Debugging},
  year   = {2026},
  url    = {https://normaluhr.github.io/en/p/trainer-generator-consistency}
}