ABSeeker: Solving Credit Assignment in Long-Horizon Search Agents

What Changed

Training long-horizon search agents has historically relied on trajectory-level supervision, where the model receives a binary reward—success or failure—only after completing a multi-step sequence of actions. These actions typically involve searching, retrieving, verifying, and integrating evidence. This approach suffers from a fundamental flaw: the credit assignment problem. When an agent fails, it is difficult to determine which specific step in the sequence caused the failure. Conversely, when an agent succeeds, it is unclear which steps were truly instrumental and which were redundant or erroneous.

Existing training paradigms, both in Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), often treat all steps within a trajectory uniformly. This leads to inefficient learning, as the model struggles to distinguish between high-quality, evidence-gathering actions and noise. The introduction of ABSeeker changes this dynamic by implementing Answer-Backtracked Credit Assignment (ABC). This framework shifts the paradigm from sparse, outcome-based rewards to dense, step-level supervision. By explicitly identifying which intermediate steps are necessary to reach a correct answer, the framework allows for more granular optimization of agent behavior.

Technical Details

The core of the ABSeeker framework lies in two distinct phases: Answer-Backtracked Clue Recovery and Clue-Anchored Step Scoring.

Answer-Backtracked Clue Recovery

Given a query and its ground-truth answer, the system performs a reverse-engineering process. It traces backward from the final answer to identify the intermediate clues or information fragments required to solve the question. This creates a dependency graph of sorts, where the final answer is anchored to specific pieces of evidence that must be retrieved during the search process.

Clue-Anchored Step Scoring

Once the intermediate clues are identified, the framework evaluates each step taken by the agent against these clues. This process converts the sparse binary outcome (did the agent get the right answer?) into a dense reward signal for every turn in the trajectory.

This dense signal is then utilized in two specific training modalities:

1. ABC-SFT: The framework reweights the loss function for each turn based on the computed step-level scores. This forces the model to prioritize learning the actions that contribute to clue discovery.
2. ABC-GRPO: The framework utilizes the step-level scores as rewards within the Group Relative Policy Optimization (GRPO) algorithm. By providing a dense reward signal at every step, the policy optimization becomes significantly more stable and efficient compared to standard trajectory-level RL.

Building on this, the researchers trained the ABSeeker model using Qwen3.5-4B as the base architecture. Remarkably, this training required only 8.5k examples, demonstrating high data efficiency in an era where massive datasets are often considered mandatory for agentic performance.

Benchmark Analysis

The performance improvements provided by the ABC framework are significant, particularly when comparing the base model against the version enhanced with context management. The following table summarizes the performance on the BrowseComp and BrowseComp-ZH benchmarks.

These results indicate that the dense supervision provided by ABC allows a 4B-parameter model to achieve performance levels typically reserved for models in the 30B-parameter range. The jump in performance when context management is enabled suggests that the model is not just learning to search, but is effectively utilizing the retrieved context to synthesize answers.

Developer Implications

For engineers working on agentic workflows, the implications of ABSeeker are twofold: data efficiency and model size optimization.

• Data Efficiency: The ability to achieve competitive results with only 8.5k examples is a major departure from the trend of requiring millions of training samples. This suggests that the quality of supervision—specifically, the granularity of the reward signal—is more critical than raw volume for search-based tasks.
• Small Model Viability: The success of the Qwen3.5-4B base model demonstrates that small, efficient models can perform complex, multi-step search tasks if the training objective is correctly aligned with the task structure. This reduces the inference cost and latency, making it feasible to deploy sophisticated search agents in resource-constrained environments.

Developers should consider implementing similar clue-anchored scoring in their own RLHF or SFT pipelines. If your agent is failing to converge on complex tasks, the issue may not be the model architecture, but rather the sparsity of the reward signal. Transitioning to a dense, step-level scoring mechanism, even if it requires a heuristic-based back-tracking approach, can provide the necessary signal for the model to learn effective search strategies.

Bottom Line

ABSeeker demonstrates that the bottleneck in training long-horizon search agents is not model capacity, but the quality of credit assignment. By moving away from trajectory-level rewards and adopting a dense, clue-anchored supervision strategy, the framework enables smaller models to punch significantly above their weight class. This approach offers a scalable path forward for building efficient, specialized search agents without the need for massive, compute-intensive training runs.

Pneumetron