RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

What Changed

For years, the primary bottleneck in scaling robot learning has been the reward model. To teach a robot a task, engineers typically rely on human-provided preferences or normalized progress metrics to define what success looks like. These methods are notoriously difficult to scale because they are tied to specific task anchors that do not transfer well across different robot embodiments or diverse data sources. The research community has struggled to find a universal recipe for learning value-related capabilities from the massive, heterogeneous corpora that have fueled breakthroughs in large language models.

RynnValue marks a departure from this paradigm. Instead of relying on subjective human preferences or task-specific progress markers, the researchers behind RynnValue use temporal distance—the directed cost-to-go from any given observation to a language-specified goal. Because temporal distance can be derived directly from the timestamps inherent in any trajectory data, the model eliminates the need for expensive, noisy, or difficult-to-scale annotation pipelines. By leveraging this intrinsic signal, RynnValue scales to over 7,000 hours of robotic manipulation data and approximately 3 million instruction-conditioned clips, effectively turning raw temporal data into a robust, general-purpose reward signal.

Technical Details

The core innovation of RynnValue lies in how it frames the value learning problem. By treating the reward as a function of temporal distance, the model learns to predict how many steps remain until a goal is achieved, conditioned on a language instruction. This formulation allows the model to ignore the specific mechanics of the robot's embodiment and focus on the progression of the state toward the goal.

To make this learning process reliable at scale, the authors implemented three specific architectural and training strategies designed to prevent the model from collapsing into trivial solutions:

1. Random Temporal Sampling: This ensures the model learns to estimate distance across various time horizons, preventing it from overfitting to short-term dynamics.
2. Temporal-Order Shuffling: By disrupting the natural sequence of clips, the model is forced to learn the underlying semantic relationship between states and goals rather than simply memorizing the sequence of frames.
3. Value-Isolation Attention: This mechanism suppresses shortcuts that would otherwise leave the model insensitive to regressions or failures, ensuring that the predicted value actually correlates with the success of the task.

Once trained, RynnValue acts as a foundation model for reward. It can be converted into dense rewards via potential-based shaping, which provides a smooth, continuous signal for policy optimization. This shaping is critical for real-world deployment, where sparse rewards often lead to inefficient exploration and policy instability.

Benchmark Analysis

The performance gains provided by RynnValue are substantial, particularly when compared to existing methods that rely on preference supervision or progress estimation. The following table summarizes the performance metrics on the RBM-EVAL-OOD benchmark, which tests the model's ability to generalize to unseen tasks and environments.

Beyond these evaluation scores, the practical impact on real-world policy success is clear. When integrated into robotic control loops, RynnValue demonstrated significant improvements in task completion rates:

• Online Policy Success: Improved from 52.5% to 72.5%.
• Offline Policy Success: Improved from 63.8% to 82.5%.

These numbers indicate that the temporal distance signal is not just a theoretical improvement but a practical tool for increasing the reliability of robotic policies in both training and deployment phases.

Developer Implications

For developers and robotics engineers, RynnValue suggests a shift in how data pipelines should be constructed. The most immediate implication is the reduced reliance on high-quality, human-labeled preference datasets. If temporal distance can be derived from existing logs, then any organization with a large repository of robot trajectory data—regardless of whether it was originally collected for reward modeling—can potentially train a high-performing value model.

This opens the door to zero-shot generalization. Because the model learns a general value function rather than a task-specific reward, it can be applied to unseen tasks, different robot embodiments, and novel viewpoints without further fine-tuning. This is a significant step toward the "foundation model" era of robotics, where a single, pre-trained reward model can serve as the backbone for a wide variety of downstream manipulation tasks.

However, developers should be aware that the quality of the temporal distance signal is still dependent on the quality of the underlying trajectory data. While the model is robust, it is not magic; it still requires diverse, high-quality data to learn the nuances of complex manipulation. The use of potential-based shaping also requires careful tuning to ensure the dense rewards do not introduce unintended biases into the policy.

Bottom Line

RynnValue demonstrates that we do not need to rely on human preferences to teach robots how to evaluate their own progress. By shifting the focus to temporal distance, the researchers have unlocked a scalable, data-efficient path for training robotic value models. With significant gains in both benchmark scores and real-world success rates, this approach provides a viable template for building generalist robot policies that can operate reliably across diverse, unseen environments.

Pneumetron