Microsoft’s rStar-Math Framework Lets Small AI Models Outperform OpenAI’s o1 Series

Microsoft has introduced rStar-Math, a continuation and refinement of its earlier rStar framework, to push the boundaries of small language models (SLMs) in mathematical reasoning.

Designed to rival larger systems such as OpenAI’s o1-preview, rStar-Math achieves remarkable benchmarks in problem-solving while demonstrating how compact models can perform at competitive levels. This development showcases a shift in AI priorities, moving from scaling up to optimizing performance for specific tasks.

Advancing from rStar to rStar-Math

The rStar framework from last summer laid the groundwork for enhancing SLM reasoning through Monte Carlo Tree Search (MCTS), an algorithm that refines solutions by simulating and validating multiple paths.

rStar demonstrated that smaller models could handle complex tasks, but its application remained general. rStar-Math builds on this foundation with targeted innovations tailored for math reasoning.

Central to rStar-Math’s success is its code-augmented chain-of-thought (CoT) methodology, where the model produces solutions in both natural language and executable Python code.

This dual-output structure ensures that intermediate reasoning steps are verifiable, reducing errors and maintaining logical consistency. The researchers emphasized the importance of this approach, stating, “Mutual consistency mirrors the common human practice in the absence of supervision, where agreement among peers on derived answers suggests a higher likelihood of correctness.”

In addition to CoT, rStar-Math introduces a Process Preference Model (PPM), which evaluates and ranks intermediate steps based on quality. Unlike traditional reward systems that often rely on noisy data, the PPM prioritizes logical coherence and accuracy, further enhancing the model’s reliability. The researchers write:

“The PPM leverages the fact that, although Q-values are still not precise enough to score each reasoning step despite using extensive MCTS rollouts, the Q-values can reliably distinguish positive (correct) steps from negative (irrelevant/incorrect) ones.

Thus the training method constructs preference pairs for each step based on Q-values and uses a pairwise ranking loss to optimize PPM’s score prediction for each reasoning step, achieving reliable labeling. This approach avoids conventional methods that directly use Q-values as reward labels, which are inherently noisy and imprecise in stepwise reward assignment.”

Finally, a four-round self-evolution recipe that progressively builds both a frontier policy model and PPM from scratch.

rSTar-Math reasoning procedure (Source: research paper)

Performance That Challenges Larger Models

rStar-Math sets new standards in mathematical reasoning benchmarks, achieving results that rival, and in some cases surpass, those of larger AI systems.

On the GSM8K dataset, a test for math reasoning, the accuracy of a 7-billion-parameter model improved from 12.51% to 63.91% after integrating rStar-Math. In the American Invitational Mathematics Examination (AIME), the model solved 53.3% of problems, placing it among the top 20% of high school participants.

The MATH dataset results were equally impressive, with rStar-Math achieving a 90% accuracy rate, outperforming OpenAI’s o1-preview.

Performance of rStar-Math and other frontier LLMs on the most challenging math benchmarks (Source: research paper)

These achievements highlight the framework’s ability to enable SLMs to handle tasks previously dominated by resource-intensive large models. By emphasizing logical consistency and verifiable intermediate steps, rStar-Math addresses one of AI’s most persistent challenges: ensuring reliable reasoning across complex problem spaces.

Technical Innovations Driving rStar-Math

The evolution from rStar to rStar-Math introduces several key advancements. The integration of MCTS remains central to the framework, enabling the model to explore diverse reasoning paths and prioritize the most promising ones.

The addition of CoT reasoning, with its focus on code verification, ensures that the outputs are both interpretable and accurate.

Perhaps most transformative is rStar-Math’s self-evolutionary training process. Over four iterative rounds, the framework refines its policy model and PPM, incorporating higher-quality reasoning data at each step.

This iterative approach allows the model to continuously improve its performance, achieving state-of-the-art results without relying on distillation from larger models.

Comparing rStar-Math to OpenAI’s o1

While Microsoft focuses on optimizing smaller models, OpenAI continues to prioritize scaling up its systems.

o1 Pro Mode, introduced in December 2024 as part of the ChatGPT Pro Plan, offers advanced reasoning capabilities tailored for high-stakes applications like coding and scientific research. OpenAI reported that o1 Pro Mode achieved an 86% accuracy rate on AIME and a 90% success rate in coding benchmarks like Codeforces.

rStar-Math represents a shift in AI innovation, challenging the industry’s focus on larger models as the primary means of achieving advanced reasoning. By enhancing SLMs with domain-specific optimizations, Microsoft offers a sustainable alternative that reduces computational costs and environmental impact.

The framework’s success in mathematical reasoning opens doors to broader applications, from education to scientific research.

The researchers plan to release rStar-Math’s code and data on GitHub, paving the way for further collaboration and development. This transparency reflects Microsoft’s approach to making high-performance AI tools accessible to a wider audience, including academic institutions and mid-sized organizations.

As the competition between Microsoft and OpenAI intensifies, the advancements introduced by rStar-Math highlight the potential of smaller models to challenge the dominance of larger systems. By prioritizing efficiency and accuracy, rStar-Math sets a new benchmark for what compact AI systems can achieve.