SIS-Bench: A New Benchmark for Self-Awareness and Spatial Cognition in UAV Embodied Intelligence

What Changed

Autonomous Unmanned Aerial Vehicle (UAV) systems are increasingly relying on Multimodal Large Language Models (MLLMs) for navigation and operation in complex environments. While existing benchmarks primarily focus on environmental understanding and spatial cognition, they often overlook the agent's self-awareness. To address this, a new benchmark, SIS-Bench, has been introduced. This benchmark provides a unified framework for evaluating embodied spatial intelligence in UAV scenarios, encompassing both the agent's understanding of its surroundings ('space') and its coherent representation of itself ('self').

SIS-Bench organizes its evaluation across two primary dimensions: 'space' and 'self,' and further categorizes tasks into a three-level hierarchy: perception, memory, and reasoning. This structured approach allows for a more granular assessment of MLLMs' capabilities in embodied intelligence. The benchmark comprises 4,856 question-answer pairs derived from 1,646 real-world UAV videos, constructed through a task-conditioned pipeline with expert verification. This comprehensive dataset aims to provide a robust foundation for evaluating MLLMs in scenarios that demand both environmental understanding and internal self-representation.

Technical Details

SIS-Bench's core innovation lies in its 'self-in-space' formulation, which explicitly integrates self-awareness into the evaluation of embodied spatial intelligence. Unlike previous benchmarks that implicitly treat self-awareness, SIS-Bench provides dedicated tasks and metrics for this dimension. The benchmark's hierarchical structure—perception, memory, and reasoning—allows for a multi-faceted assessment of cognitive abilities. Perception tasks might involve identifying the UAV's current state or immediate surroundings, memory tasks could test the recall of past trajectories or environmental features, and reasoning tasks might require inferring future actions or understanding complex spatial relationships involving the UAV itself.

The dataset's construction involved extracting 4,856 question-answer pairs from 1,646 real-world UAV videos. This process utilized a task-conditioned construction pipeline, ensuring that the questions are relevant to practical UAV operations and validated by experts. This methodology aims to create a realistic and challenging evaluation environment for MLLMs. The benchmark's design specifically targets the limitations observed in current MLLMs, particularly their struggle with dynamic and agent-centered processes. To mitigate these limitations, the research also explored a motion-aware representation that integrates self-related dynamics through optical flow and visual feature fusion. This approach aims to provide MLLMs with a richer understanding of their own movement and its impact on their perception and interaction with the environment.

Benchmark Analysis

Extensive evaluations using SIS-Bench have revealed fundamental limitations in current MLLMs when modeling dynamic and agent-centered processes. A significant finding is the clear imbalance between spatial cognition and self-awareness, indicating that MLLMs generally perform better at understanding the external environment than at maintaining a coherent representation of their own state and actions. Furthermore, the evaluations showed a progressive performance degradation across cognitive levels, meaning MLLMs' performance tends to decrease as tasks move from perception to memory and then to reasoning.

Specifically, the integration of a motion-aware representation, which incorporates self-related dynamics through optical flow and visual feature fusion, consistently improved performance. This improvement was observed not only in spatial cognition tasks but also in self-awareness tasks, and it generalized to downstream UAV decision-making tasks. This suggests that explicitly modeling agent motion can significantly enhance both the understanding of space and the awareness of self within that space for MLLMs operating in embodied scenarios.

Developer Implications

The introduction of SIS-Bench provides a critical tool for developers working on autonomous UAV systems. By offering a structured and comprehensive benchmark that explicitly evaluates self-awareness alongside spatial cognition, developers can now identify specific weaknesses in their MLLMs. The observed imbalance between spatial cognition and self-awareness, and the performance degradation across cognitive levels, indicates areas where current models require significant improvement. This benchmark can guide the development of more robust and reliable MLLMs for UAVs.

The findings regarding the benefits of motion-aware representations are particularly relevant for model architects. Incorporating self-related dynamics through optical flow and visual feature fusion can be a viable strategy to enhance MLLM performance in embodied scenarios. This suggests a shift towards models that not only process environmental data but also integrate their own kinematic and dynamic states more deeply into their representations. Developers can leverage this insight to design MLLMs that are more attuned to their own actions and their impact on the environment, leading to more intelligent and safer autonomous systems.

Bottom Line

SIS-Bench represents a significant step forward in evaluating embodied intelligence for autonomous UAVs. By explicitly addressing the often-overlooked dimension of self-awareness, it provides a more holistic assessment of MLLMs' capabilities. The benchmark highlights current MLLM limitations in dynamic, agent-centered processes and reveals a performance gap between spatial cognition and self-awareness. The research also offers a promising direction by demonstrating that motion-aware representations can consistently improve both spatial cognition and self-awareness, ultimately benefiting UAV decision-making. This work underscores the importance of self-awareness for advancing embodied spatial intelligence and provides both a new evaluation tool and empirical evidence for future model development.

Pneumetron