Paper review · November 13, 2026
Scaling mathematical proof training data with structured textbooks (TOME-Math)
Scaling Mathematical Proof Supervision from Structured Textbooks
Converting 2,118 math textbooks into 380,000 proof QA items that preserve chapter order and dependencies between statements raised a 30B model's average across six benchmarks by 16.7 percentage points.
Problem
Math training data is usually built as independent 'problem→solution' pairs. In the process, textbook structure such as chapter order, the link between a theorem and its proof, and references to earlier statements is lost. The authors point out that there is still no clear way to scale up natural-language proof training data while preserving this structure.
Method
DeepSeek-OCR converts PDFs to Markdown, and gpt-oss-120b together with a state machine reconstructs the table of contents and chapter boundaries. Gemini-family models extract theorems, lemmas and definitions along with their proofs to build Proof QA (205,861 items). Reasoning steps omitted within proofs (gaps) are identified and rewritten as standalone statements with proofs to build Gap QA (174,431 items). Each record carries book, chapter and source-location information plus referenced statements, and an LLM judge checks quality along five axes (completeness, correctness, LaTeX formatting, self-containment and atomicity). The resulting data is used for SFT (supervised fine-tuning) of Qwen3-30B-A3B-Base, and the effects of structure-based retrieval and a textbook-order curriculum are tested separately.
Results
- The simple average across six benchmarks (TOME-LongForm, TOME-Objective, MiniF2F, ProofNet, MATH-500, AIME 2026) rose from 29.17% for the base model to 45.83%, higher than the public SFT datasets MathInstruct (41.10%), MetaMath (34.64%) and OpenMath1 (29.13%).
- It ranked first on 5 of the 6 benchmarks. On AIME 2026 it scored 10.19%, second after MathInstruct (11.98%).
- It outperformed OpenMath1 (1.82 million items), about 4.8 times larger, on all six benchmarks, which the authors interpreted as showing that source and format matter more than data volume.
- Without changing parameters, retrieving supporting material via textbook structure and adding it to the context raised TOME-LongForm pass@1 from 13.25% (no context) to 17.38% (statements + Gap QA). Random retrieval from the same textbook with the same token budget reached 14.00%.
- The authors reported that, across six subjects and three data scales, a curriculum that trains in textbook order was consistently better than random or reverse order.
Limitations
As the authors acknowledge, the comparison with public datasets mixes several factors and cannot isolate the effect of 'textbook structure' alone. Data extraction, gap generation and quality checks were all done by LLMs, and TOME-LongForm grading also relies on an LLM verifier. Only 19% of generated Gap QA candidates were kept, indicating high variance in generation quality. Experiments used only one base model, Qwen3-30B-A3B. The input text was cut off partway, so detailed numbers for the curriculum experiments could not be checked.
Why it matters
It provides evidence that how data is organized may also be an important variable when building math reasoning data. It can serve as a reference when designing pipelines that turn structured documents, such as textbooks or internal company manuals, into training data. The authors said they plan to release the corpus.
Key terms
- Gap QA
- Training data that rewrites reasoning steps omitted or left implicit in textbook proofs as standalone statements, each with its own proof.
- Curriculum learning
- A training approach that presents training data from easier to harder material (here, in textbook chapter order) instead of shuffling it randomly.
- SFT(Supervised Fine-Tuning)
- A fine-tuning method that further trains a pretrained model on input-output pairs with correct answers.
- pass@1
- The probability that a single answer generated by the model is correct. It is estimated by sampling multiple times and averaging.
Original
Authors: Youqing Fang, Zhouqi Hua, Yinhao Tang, Yuzhe Gu, Lingkai Kong, Bin Liu, Weiming Zhang, Kai Chen, et al. · Posted 2026-10-29 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.