Reflect, rewrite, repeat: How simple arithmetic enables advanced reasoning in small language models

Contemporary advancements in language model reasoning often depend on computationally intensive reinforcement learning (RL) and massive datasets, creating prohibitive barriers for resource-constrained teams. In this work, we demonstrate that high-quality, iterative training on minimal data can rival modern RL approaches. We introduce a resource-efficient framework that combines Direct Preference Optimization (DPO) and Supervised Fine-Tuning (SFT) with selective guidance from larger models, iteratively refining solutions through a 'reflect, rewrite, repeat' - the R3 cycle. Using Qwen-2.5-7B and Qwen 2.5-Math-7B as base models, our method shows meaningful performance improvements across arithmetic, symbolic and cognitive reasoning benchmarks —including GSM8K (83.1% → 88.6%), AIME'25@10 (20.0% → 30.0%) and LastLetterConcat (40.7% → 53.3%) problems. The model-agnostic nature of our R3 pipeline is further demonstrated through substantial improvements when applied to Mistral and LLaMA-based models. Remarkably, these gains are achieved using only 700 basic arithmetic training samples—a stark contrast to the hundreds of thousands of examples typically required by RL-based systems. Our results suggest that reasoning improvements need not strictly depend on large-scale data. By emphasizing strategically curated training grounded in foundational principles, we achieve competitive generalization and efficiency with minimal resource overhead. An additional benefit of R3 is the production of high-quality SFT data with high-fidelity reasoning traces. This work highlights the viability of data-centric approaches for advancing reasoning capabilities, providing both a practical framework for resource-limited teams and a scalable source of tailored SFT data to enhance model performance through structured reasoning patterns.