Paper review · November 11, 2026
The cost hidden by practice accuracy: worked examples in self-generated training data and compositional use of new rules
A Cost That Practice Accuracy Hides: Worked Solutions in Self-Generated Training Data and the Compositional Use of New Rules
Comparing models by whether worked examples were shown when creating self-generated training data, the authors show that even with the same practice accuracy, the ability to reuse new rules in sequence differs greatly.
Problem
A widely used approach is to have a model solve practice problems with the rules in context, keep only correct outputs, and fine-tune on them without the rules. Providing program-generated worked examples at this stage shortens the training targets and lowers cost. However, looking only at accuracy on the practiced operation can miss differences in the ability to reuse the new rules in other ways.
Method
The authors defined 2 newly created quantitative rules (voltage relaxation T1 and current-dependent available capacity T2) for a fictional battery cell, "Kestrel." A 2B-parameter reasoning model creates training targets for 128 forward problems either by seeing only the rule sheet (Gen) or by seeing the sheet together with program-written solutions (Inst). Only targets with correct final answers are kept, and training uses LoRA+ with the context removed. Evaluation without context is split into practiced forward use, untrained inversion, and composition, which applies the same rule across two consecutive intervals. The experiments also include a pre-registered prospective validation, tests on new argument values, an independently sampled second target bank, and a 10-epoch training condition that matches training volume.
Results
- After 6 training epochs, in the prospective validation, targets made from the sheet alone had a 24.3 percentage point higher success rate on composition problems (95% confidence interval 14.6–34.0).
- On argument values not seen in training, forward accuracy was the same for both at 87.5%, but composition accuracy was 58.3% versus 23.6%, a 34.7 percentage point gap (25.7–43.8).
- The same direction was reproduced on an independently sampled second target bank (+14.6 percentage points, 2.1–26.0). When the worked-example policy was trained for 10 epochs to match the number of domain tokens, the gap shrank to 16.3 percentage points, and on the second bank it was 6.2 percentage points (−6.3–18.8), which did not pass the pre-specified test.
- The worked-example policy used 42% fewer capture tokens and had 28% shorter update time. However, the resulting model generated 2.6x more tokens per successful composition (2.1x on the second bank).
Limitations
The experiment uses one fictional domain, one main model (2B) and one training recipe. Composition covers only sequential reuse of the same rule, not combining two rules. The authors could not isolate which property of the targets causes the difference. Matching training volume shrinks the gap, and it is not significant on the second bank, so the size of the difference is uncertain. The answer-only condition did not pass the forward accuracy threshold, so no conclusion could be drawn for it.
Why it matters
When running a self-generated data pipeline, the metrics commonly logged (generation cost, training time, accuracy on similar problems) are not enough to choose a context design. The study provides evidence that uses not required in training, as well as inference cost, should also be evaluated.
Key terms
- Capture context
- Material placed in the model's context when it generates its own training targets. It is removed during training and evaluation. Here, the study compared giving only a rule sheet with giving the sheet plus worked examples.
- Context Distillation
- A method that trains a model to produce, without the extra context, the same output it generated while seeing that context.
- Underspecification
- The phenomenon where the same pipeline can produce models that perform equally on validation data from the training distribution but behave differently outside that distribution.
Original
Authors: Jiantong Xu · Posted 2026-11-09 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.