Paper review · November 6, 2026

GIST: Gibbs-Target-Based Hierarchical Data Selection for Offline RLVR

GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR

Language models Reinforcement learning Advanced cs.AI

The authors propose GIST, which selects the responses and tokens to train on by analyzing two-way mismatch with the Gibbs optimal policy of KL-regularized RLVR. It reduced supervised tokens by 66.6% while improving accuracy.

Problem

Offline RLVR (reinforcement learning with verifiable rewards) trains on a pool of pre-sampled responses and usually supervises every token of every response. But responses and tokens differ in how much information they carry. Existing methods handled only one of response selection or token selection, and did not address which tokens are useful depending on correctness and sampling method.

Method

Theory: The authors measure the gap between the current policy and the Gibbs target policy, which is the optimal solution of KL-regularized RLVR, using bidirectional KL, and derive a KL curvature score that bounds this gap from above and below. At the reference policy, this mismatch decomposes exactly. For correct responses, the forward term dominates and information concentrates in low-probability tokens. For incorrect responses, the reverse term dominates and information concentrates in frequently generated responses and high-probability tokens. Method: For each prompt, 16 responses are generated at temperature 1.0 and 16 at 0.3. Four correct responses are selected from the temperature-1.0 pool and four incorrect responses from the 0.3 (near-mode) pool, and only prompts that yield both kinds are kept. Only two kinds of tokens are supervised: correct-response tokens whose initial-policy probability is below the response average, and incorrect-response tokens whose probability is above it. All selection is done once before training, and it was applied to the BOLT (weighted likelihood) and SFT objectives.

Results

Across three models (Qwen3-0.6B, Qwen3.5-2B, Gemma-4-E2B-it) and two objectives (BOLT and SFT), results on GSM8K, MATH-500 and MBPP improved in 29 of 30 settings, and average accuracy rose in every case.
On Qwen3-0.6B/GSM8K, Strict accuracy was higher than that of seven selection baselines trained on the same candidate pool. The number of supervised tokens was 66.6% lower than training on all candidates for the same prompts, and 31.5% lower than random selection.
In the same comparison, it also used 49.9% fewer optimizer steps than training on all candidates.

Limitations

The quotas for correct and incorrect responses (four each) are a practical choice, not values derived from the theory. As a result, the hardest prompts, which rarely produce a correct answer, are left out of training. The theory does not determine the relative weight of the two classes, so the same quota was used for both. The analysis concerns the mismatch at the reference policy and does not guarantee the outcome of training on the selected data (noted by the authors). Experiments were limited to small models of 2B parameters or fewer and to math and code benchmarks.

Why it matters

It offers a principled criterion for deciding 'what to keep and what to supervise' in RLVR training data. Because it uses only signals available before training, such as temperature and initial-policy probability, it is cheap to implement and can improve performance while greatly reducing training tokens and compute.

Key terms

RLVR(Reinforcement Learning with Verifiable Rewards): Reinforcement learning that uses rewards whose correctness can be judged automatically, such as checking a math answer or running code tests.
Gibbs optimal policy: The solution to the KL-regularized reward maximization problem: a distribution formed by multiplying the reference policy's probabilities by an exponential weight of the reward. It is the target that offline training tries to match.
Forward/reverse KL: Forward KL measures how much of the behavior the target wants the policy misses; reverse KL measures how much probability the policy places where the target does not want it.
Temperature pairing: GIST's response selection method: it picks correct responses from temperature-1.0 samples and incorrect responses from low-temperature (near-mode) samples to get the most informative response of each kind.

Original

Authors: Xuliang Huang, Tianxiang Zhao, Chengwei Qin, Yao Shu · Posted 2026-11-04 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.