DRACO Solves Long-Horizon Credit Assignment Without Verifiable Rewards

What Changed

Training long-horizon AI agents has historically relied on the availability of verifiable rewards—programmatic checkers that can score an agent's output at every step. However, most real-world agentic tasks, such as complex software engineering or multi-step reasoning, lack these ground-truth signals. When verifiers are absent, developers often resort to multi-criteria rubrics scored once per trajectory. This approach suffers from a fundamental flaw: a single scalar reward is an insufficient signal for a process spanning dozens or hundreds of steps.

DRACO (Distributing Rubric-based Advantage for Credit Optimization) addresses this by transforming how agents interpret sparse feedback. Instead of treating a trajectory as a monolithic success or failure, DRACO dynamically generates rubrics during training to track the policy's evolving capabilities. It then redistributes the trajectory-level judgment across the specific steps that contributed to those outcomes. This creates differentiated, per-step advantages for GRPO (Group Relative Policy Optimization), allowing the model to learn which specific actions were responsible for success or failure without requiring a trained attribution module.

Technical Details

The core innovation of DRACO lies in its closed-form redistribution mechanism. In standard reinforcement learning setups, credit assignment is often noisy when rewards are sparse. If an agent completes a 50-step task and receives a single score, the model struggles to identify which of those 50 steps were optimal.

DRACO operates in the outcome-blind setting, meaning it does not have access to ground-truth success signals. The process functions as follows:

1. Dynamic Rubric Generation: As the policy evolves, DRACO generates rubrics that adapt to the agent's current capabilities. This ensures the evaluation criteria remain relevant as the agent gets smarter.
2. Trajectory Scoring: Once a full trajectory is completed, the rubric is applied to produce a score. This is a standard practice, but where DRACO diverges is in the next step.
3. Advantage Redistribution: Rather than applying the scalar score to the entire trajectory, DRACO redistributes this judgment. It mathematically maps the rubric score back to the specific steps responsible for the annotated rubric items. This produces a fine-grained advantage signal for each step.
4. GRPO Integration: These per-step advantages are fed directly into the GRPO framework. Because the redistribution is closed-form, it avoids the overhead and potential instability of training a separate attribution module (like a learned reward model or a Q-function) to estimate credit.

This approach effectively solves the credit assignment problem by creating a synthetic, fine-grained reward structure that aligns with the agent's actual trajectory, all without needing an external verifier or a complex, trainable critic.

Benchmark Analysis

The researchers evaluated DRACO on two primary benchmarks: AppWorld and Tau-Bench. In both cases, the method demonstrated significant gains over baseline models and standard GRPO implementations.

On the AppWorld benchmark, DRACO achieved a 15.9-point improvement over the base model and a 5.3-point improvement over GRPO trained with a sparse ground-truth reward. Notably, it achieved these results without using any programmatic verifiers. On the out-of-domain Tau-Bench, DRACO gained 5.3 points over the base model, outperforming both ground-truth-reward training and other existing rubric-based training methods.

Developer Implications

For engineers building agentic systems, DRACO represents a significant shift in how we handle reward signals. The primary takeaway is that you no longer need a perfect programmatic verifier to achieve high-performance fine-tuning. If you can define a multi-criteria rubric—which is often easier than writing a full test suite for every possible agent action—you can now derive fine-grained training signals.

This method is particularly useful for:

• Complex Reasoning Tasks: Where the path to the solution is long and non-linear.
• Resource-Constrained Environments: Since the redistribution is closed-form, it does not require the heavy compute overhead associated with training auxiliary reward models or value functions.
• Iterative Development: Because the rubrics are dynamic, the training process scales with the model's capability, reducing the need to manually update reward functions as the agent improves.

Developers should note that while DRACO is effective, it still relies on the quality of the rubrics provided. The dynamic generation helps, but the initial rubric definition remains a critical design choice.

Bottom Line

DRACO provides a robust, computationally efficient method for training agents in environments where ground-truth rewards are unavailable. By mathematically redistributing trajectory-level scores into per-step advantages, it bridges the gap between sparse feedback and fine-grained policy optimization. For teams struggling with credit assignment in long-horizon tasks, this approach offers a clear path toward better performance without the burden of maintaining complex verifier infrastructure.

Pneumetron