Evidence-Backed Video Question Answering: Bridging Reasoning and Visual Grounding in Video LLMs

What Changed

Traditional Video Large Language Models (Video LLMs) have demonstrated proficiency in question answering (QA) tasks but frequently lack transparency, delivering textual responses without explicit visual grounding. Existing methods for explainability, such as textual rationales or sparse bounding boxes, often fall short in capturing the intricate dynamics of video content, including occlusions and non-rigid deformations. This limitation makes it challenging to verify the visual basis of a model's answer.

To address this, researchers have proposed Evidence-Backed Video Question Answering (E-VQA). E-VQA is a novel task that mandates models to not only generate a semantic answer but also to provide precise spatio-temporal evidence. This evidence includes temporal segments and dense, tracked object segmentation masklets. The goal is to move beyond black-box operations, enabling verifiable visual grounding for Video LLM outputs.

Accompanying E-VQA, the ST-Evidence benchmark has been introduced. This is the first human-verified benchmark designed for both discriminative and generative pixel-level grounding. Its creation facilitates the evaluation of models' abilities to provide fine-grained visual evidence. Furthermore, a 160k-scale dataset, ST-Evidence-Instruct, was developed using scalable, automated generation pipelines. This dataset is designed to bridge high-level reasoning with fine-grained grounding, providing a robust resource for training models in this new paradigm.

Technical Details

The core innovation of E-VQA lies in its requirement for joint output: a semantic answer and precise spatio-temporal evidence. This evidence is defined as temporal segments, indicating when an event or object is relevant, and dense, tracked object segmentation masklets, which provide pixel-level localization and tracking of objects throughout the video. This granular level of detail is intended to overcome the limitations of sparse bounding boxes, which can struggle with complex visual phenomena.

To support the E-VQA task, the ST-Evidence benchmark was developed. This benchmark is unique in its human-verified nature for pixel-level grounding, ensuring high-quality annotations for both discriminative (identifying existing evidence) and generative (creating new evidence) tasks. The benchmark allows for a more rigorous evaluation of a model's true visual perception, distinguishing it from mere QA accuracy.

Initial evaluations of state-of-the-art models on ST-Evidence revealed a significant decoupling between their QA accuracy and their actual visual perception capabilities. This suggests that simply scaling up existing Video LLMs does not inherently improve their ability to provide verifiable visual grounding. To mitigate this, the ST-Evidence-Instruct dataset was created. This dataset, comprising 160,000 examples, was generated through scalable, automated pipelines, ensuring a large and diverse set of training data that links high-level reasoning with fine-grained visual grounding.

Fine-tuning grounded Video LLMs on the ST-Evidence-Instruct dataset yielded substantial performance improvements. For instance, a 7B model fine-tuned on this data showed gains of +27.2 t-mean and +13.8 J&F over corresponding size-matched UniPixel baselines. These metrics indicate an enhanced ability to both identify the correct temporal segments (t-mean) and accurately segment and track objects (J&F, which combines Jaccard index and F-measure for segmentation and tracking quality).

The methodology emphasizes a shift from purely textual outputs to multimodal outputs that integrate language with precise visual evidence. This approach aims to build more explainable AI systems by making the reasoning process visually transparent and verifiable.

Benchmark Analysis

Evaluations of state-of-the-art models on the ST-Evidence benchmark highlighted a critical observation: a decoupling between Question Answering (QA) accuracy and true visual perception. This implies that models might provide correct answers without genuinely understanding the visual context, operating more as black boxes.

Fine-tuning grounded Video LLMs on the ST-Evidence-Instruct dataset resulted in notable performance improvements. Specifically, a 7B model, when fine-tuned, demonstrated substantial gains over size-matched UniPixel baselines. The improvements observed were +27.2 t-mean and +13.8 J&F. The t-mean metric typically assesses temporal localization accuracy, while J&F combines the Jaccard index (for segmentation quality) and F-measure (for tracking quality), indicating improved spatio-temporal grounding.

These numerical results establish a robust baseline for explainable, evidence-backed video understanding, demonstrating that targeted training on datasets like ST-Evidence-Instruct can effectively bridge the gap between high-level reasoning and fine-grained visual grounding.

Developer Implications

For developers working with Video LLMs, E-VQA presents a new paradigm for building more transparent and reliable systems. The availability of the ST-Evidence benchmark and the ST-Evidence-Instruct dataset provides concrete resources for training and evaluating models that can offer verifiable visual grounding.

Developers can leverage these resources to fine-tune their existing Video LLMs or develop new architectures capable of jointly outputting semantic answers and precise spatio-temporal evidence. This capability is particularly valuable in applications where trust and explainability are paramount, such as autonomous systems, surveillance, medical diagnostics, or content moderation, where understanding why a model made a certain decision is as important as the decision itself.

The shift towards dense, tracked object segmentation masklets as evidence means developers will need to integrate more sophisticated computer vision techniques, beyond simple bounding boxes, into their Video LLM pipelines. This could involve advancements in real-time object tracking and segmentation, potentially requiring more computational resources but yielding significantly richer and more accurate explanations.

The open-sourcing of code and data at https://github.com/SalesforceAIResearch/EVQA lowers the barrier to entry for developers to experiment with and implement E-VQA. This allows for direct application and further research into improving the explainability and visual perception of Video LLMs.

Bottom Line

Evidence-Backed Video Question Answering (E-VQA) represents a significant step towards making Video LLMs more transparent and trustworthy. By requiring models to provide precise spatio-temporal evidence alongside their answers, E-VQA addresses the black-box nature of current systems and enhances their visual grounding capabilities. The introduction of the human-verified ST-Evidence benchmark and the large-scale ST-Evidence-Instruct dataset provides the necessary tools for training and evaluating models in this new paradigm. The observed performance gains from fine-tuning demonstrate the efficacy of this approach in bridging the gap between high-level reasoning and fine-grained visual perception. This work establishes a robust baseline for explainable AI in video understanding, paving the way for more reliable and verifiable AI applications.