Beyond Accuracy: Measuring Latent Reasoning in Low-Resource Languages

What Changed

For years, the machine learning community has relied on accuracy benchmarks as the primary yardstick for model performance. A new study examining frontier mixture-of-experts (MoE) models—specifically those from Alibaba, OpenAI, and NVIDIA, each with 3.6 to 4.0 billion active parameters—challenges this reliance. The research demonstrates that when adapting models to a low-resource language like Greek, accuracy benchmarks are essentially noise. The researchers found that simply changing the random seed could shift accuracy scores by 7.7 points, a variance that dwarfs the actual improvements gained from data and training recipes.

The most significant finding is that base models, even when prompted in a specific language, do not actually "think" in that language. Analysis of 1,000 reasoning traces showed that base models performed zero reasoning in Greek, even when the input was in Greek. Instead, they performed their internal reasoning in English and translated the final answer. This creates a critical auditability gap: the user cannot verify the reasoning process because it occurs in a language they cannot read. The study confirms that Supervised Fine-Tuning (SFT) effectively forces the model to reason in the target language, but it fails to address fundamental structural defects.

Technical Details

The researchers utilized four MoE checkpoints to test their hypothesis. The training pipeline consisted of two distinct phases: SFT and Reinforcement Learning with Verifiable Rewards (RLVR).

SFT was successful in shifting the latent reasoning process. After fine-tuning, every checkpoint reasoned in the language of the question on approximately 98% of items. In one model family, this was achieved with 3x fewer tokens, indicating that the models became more efficient at generating reasoning traces in the target language. However, SFT proved incapable of fixing three specific defects:

1. Format Skipping: A quarter of the answers failed to adhere to the requested output format.
2. Reasoning Leakage: The final answer frequently leaked into the reasoning channel.
3. Instruction Following: An explicit instruction to "think in English" was obeyed less than 50% of the time.

To address these, the team implemented RLVR, pre-registered before training. This approach introduced verifiable rewards that penalized format violations and leakage. The results were stark: RLVR reduced format failures from 24% to 2.5% and eliminated reasoning leakage entirely (from 3.5% to 0.0%).

Furthermore, the researchers proposed six behavioral dimensions to measure these changes. Crucially, they gated these metrics to reject any measurement that correlated with output length, a common pitfall in automated evaluation. They reported that their initial instruments were flawed, failing six times before being corrected by a control group.

Benchmark Analysis

The following table illustrates the performance shifts observed during the transition from SFT to RLVR, highlighting the necessity of reinforcement learning for structural integrity.

As noted, the accuracy benchmarks themselves were found to be unreliable. The 7.7-point swing caused by random seed variation suggests that at this scale, accuracy is a poor proxy for linguistic adaptation.

Developer Implications

For engineers working on language adaptation, this paper serves as a warning against over-reliance on standard benchmarks. If your evaluation metrics correlate with output length, they are likely providing false positives.

Developers should prioritize the following:

• Auditability: Ensure that the reasoning trace is generated in the same language as the user input. If the model is "thinking" in English while outputting in Greek, you cannot audit the reasoning process for hallucinations or logic errors.
• Verifiable Rewards: SFT is insufficient for production-grade output. If your application requires strict formatting (e.g., JSON output, specific XML tags), you must implement RLVR or similar constrained decoding strategies.
• Control Groups: Always run a control group when measuring behavioral changes. As the researchers discovered, their own instruments lied to them until they introduced controls to filter out length-based correlations.

Bottom Line

Accuracy is not the same as fluency or reasoning capability. This research proves that while SFT can shift the language of reasoning, it is not a cure-all for model defects. The combination of SFT for linguistic adaptation and RLVR for structural compliance is the current gold standard for deploying reliable, multilingual reasoning models. Developers must move beyond accuracy scores and develop behavioral metrics that truly capture how the model is reasoning, not just what it outputs.

Pneumetron