ClawGym II: Solving the Black-Box Bottleneck in Agent Reinforcement Learning

What Changed

For developers building autonomous agents, the gap between model training and real-world execution has long been bridged by agent harnesses—the software environments that allow LLMs to interact with tools, file systems, and APIs. While these harnesses have become essential for long-horizon tasks, they have historically been treated as 'black boxes' during the training process. This opacity makes traditional Reinforcement Learning (RL), which typically requires differentiable access to the environment, nearly impossible to apply effectively.

ClawGym II fundamentally changes this dynamic. By introducing a unified framework that decouples policy optimization from the execution environment, the researchers have enabled stable, scalable RL training on top of complex, non-differentiable harnesses. Instead of trying to force the harness to be differentiable, the system treats the entire interaction loop as a data-generation process, capturing model calls via a serving proxy and reconstructing them into structured data. This allows developers to apply advanced RL techniques like PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization) to agents that were previously limited to supervised fine-tuning or simple prompting strategies.

Technical Details

The architecture of ClawGym II is designed to solve the instability inherent in long-horizon agent training. The pipeline consists of three primary technical innovations:

1. Sandbox-based Execution Infrastructure: The framework isolates task environments and harnesses within temporary, ephemeral sandboxes. This is critical for large-scale concurrent rollouts, ensuring that one agent's interaction does not pollute the state of another, while simultaneously preventing the resource contention that often plagues agent training.

2. Trajectory Reconstruction via Prefix Trees: Because agent harnesses are opaque, the model's interaction history is often fragmented. The framework places a serving proxy at the model boundary to intercept all calls. These calls are then organized into prefix trees. This structure allows the training algorithm to understand the branching nature of agent decision-making—where one action leads to multiple potential future states—and optimize the policy over these recovered trees rather than just linear sequences.

3. Mix-Harness Training: Perhaps the most versatile feature is the ability to jointly optimize a single model across heterogeneous harnesses. By normalizing the interaction data, the framework allows a model to learn from diverse environments (e.g., coding environments, web browsers, and file systems) simultaneously. This prevents overfitting to a single harness and promotes the development of more generalized agentic capabilities.

By adapting both critic-based PPO and critic-free GRPO to operate on these prefix trees, the framework maintains training-inference consistency. This ensures that the policy being optimized during the RL phase behaves identically when deployed in the production harness, a common point of failure in previous agent training attempts.

Benchmark Analysis

The effectiveness of this black-box RL approach is demonstrated using the Qwen3-30A3B model on the ClawGym-Bench. The results highlight the stability and performance improvements gained by moving from standard prompting or supervised fine-tuning to this RL-based harness optimization.

These gains are not merely isolated to the primary benchmark. The framework has shown consistent performance improvements on more challenging, multi-step environments such as JobBench and OfficeQA. The stability over 200-400 optimization steps is particularly notable, as RL for agents is notorious for collapsing or diverging quickly when the reward signal is sparse or the environment is noisy.

Developer Implications

For engineers working on agentic workflows, ClawGym II signals a shift away from 'prompt engineering as the primary optimization method.' We are moving toward a paradigm where the harness itself becomes a training signal.

• Infrastructure Overhead: Adopting this framework requires setting up a robust, sandboxed execution environment. This is not a 'plug-and-play' solution; it requires careful engineering of the serving proxy to ensure that the model boundary is correctly identified and that latency during training rollouts remains manageable.

• Data Efficiency: By using prefix trees to reconstruct trajectories, developers can extract more signal from fewer rollouts. This is a significant advantage for teams with limited compute budgets who cannot afford millions of environment interactions.

• Model Agnostic Potential: While the paper tests Qwen3-30A3B, the black-box nature of the framework suggests that it could be applied to any LLM capable of function calling. This could allow teams to fine-tune smaller, more efficient models to perform at the level of larger, general-purpose models by specializing them through harness-specific RL.

Bottom Line

ClawGym II moves agent development closer to a rigorous engineering discipline. By successfully applying RL to opaque, complex harnesses, it provides a viable path for training agents that are not just good at following instructions, but are capable of navigating long-horizon, multi-step tasks with high reliability. The ability to optimize across heterogeneous harnesses suggests that the future of agent training lies in unified, scalable infrastructure rather than bespoke, task-specific solutions.

Pneumetron