Local Perception and Recurrence: A New Path for Visual Reasoning Generalization

What Changed

Traditional computer vision models process entire images in a single, global computation, a stark contrast to the human visual system's sequential, foveated glimpses. Recent research, inspired by length generalization issues in language models, reveals that this global processing in vision models leads to a critical failure in generalizing to tasks with increased complexity or length beyond their training distribution. The core finding is that strictly local, sequential perception, when combined with recurrence, enables vision models to mitigate these generalization failures, offering a more robust approach to visual reasoning.

Historically, the assumption has been that simply making a model recurrent would address state-tracking and generalization issues. However, this work demonstrates that recurrence alone is insufficient if the model still maintains a global view of the input. The key insight is that locality – restricting the model to small, foveated glimpses – is a necessary ingredient. When a recurrent network is fed the entire image, it still learns "global shortcuts" that work within the training distribution but fail when faced with out-of-distribution complexity. This mirrors similar challenges observed in large language models (LLMs) on tasks requiring sequential state tracking.

Technical Details

The research investigates the behavior of vision models on simple visual puzzles designed to require the aggregation of local information across an image, such as reading switches, navigating, and tracking state. These puzzles were constructed to allow for testing length generalization by increasing their complexity at test time compared to training.

Experiments were conducted using state-of-the-art Vision-Language Models (VLMs) like GPT-5.4, Claude Sonnet 4.6, and Qwen. These global models performed well on short puzzles but exhibited a significant performance drop when the puzzles became longer. In contrast, a small recurrent agent, specifically one employing strictly local perception, demonstrated extrapolated performance at test time, maintaining accuracy on longer sequences.

To isolate the impact of locality, the researchers tested recurrent LSTM networks under three different visual input conditions:

• Global: The LSTM received the entire image at once. This configuration collapsed on out-of-distribution lengths.
• Local + Global: The LSTM received small, high-resolution crops in addition to a global view. This setup also collapsed, indicating that the presence of a global view still allowed for shortcut learning.
• Foveated: The LSTM received only small, local glimpses, with no global view. This configuration successfully generalized out-of-distribution.

This demonstrates that locality is not merely an enhancement but a necessary condition for robust generalization in recurrent vision models. Further experiments explored the role of the backbone architecture. When local glimpses were fixed, strict recurrent networks (LSTM, GRU, RNN) generalized, while Transformers, Mamba, and xLSTM architectures still degraded. This suggests that the recipe for robust compositional generalization is a combination of both local perception and a recurrent architecture; neither is sufficient on its own, but together they are.

An important trade-off was identified: small, low-resolution glimpses promote generalization but can be slow for exploration, while big, high-resolution glimpses facilitate exploration but invite global shortcuts. The study showed that with appropriate glimpse settings, a FoveAgentLSTM could maintain accuracy at resolutions well beyond its training data, a capability global models lacked.

The research also differentiated between task types. For state-tracking tasks, recurrence and locality were beneficial. However, for pure recall tasks, global VLMs outperformed local agents, indicating that the benefits of this approach are task-dependent, mirroring similar distinctions in language models.

Finally, the findings were applied to a real-world task: reasoning over mathematical plots to find a function's roots. At the same visual-compute budget, a foveated Qwen model achieved a +29 percentage point increase in accuracy (approximately 100% improvement) over a global baseline. Uniformly increasing resolution by 10x in the global model yielded only a +3.8 percentage point improvement, reinforcing the conclusion that how visual compute is spent is more critical than the sheer amount of compute.

Benchmark Analysis

• Synthetic Visual Puzzles (Length Generalization): State-of-the-art VLMs (GPT-5.4, Claude Sonnet 4.6, Qwen)