MMDiff: A New Framework for Steering Multimodal LLMs via Feature-Level Control

What Changed

Multimodal Large Language Models (MLLMs) have rapidly advanced, yet our ability to interpret their internal decision-making processes remains limited. While sparse autoencoders (SAEs) have successfully decomposed hidden states into interpretable features in pure language models, applying these techniques to multimodal systems has proven difficult. The primary challenge lies in isolating which features are truly specific to the multimodal training process versus those inherited from the base language model.

MMDiff, a new framework introduced by researchers at the University of Oxford, addresses this by implementing a "model-diffing" approach. Instead of treating the multimodal model as a black box, MMDiff trains multimodal SAEs and compares them against SAEs trained on the base language model. This allows developers to isolate features that are uniquely altered or created by multimodal training. By turning these SAEs into feature-level interfaces, the framework enables researchers to discover, audit, and causally control specific multimodal behaviors, such as visual-spatial reasoning and safety-related responses, without disrupting general performance.

Technical Details

The core of MMDiff relies on a three-pronged methodological approach designed to bridge the gap between interpretability and control:

1. Feature Isolation: The framework performs a "diff" between a base-LM SAE and a multimodal-adapted SAE. This process effectively filters out features inherent to the language-only backbone, leaving behind a set of features that are specifically responsive to the multimodal training phase.

2. Task-Specific Feature Detection: Once the multimodal-specific features are isolated, the researchers employ per-token contrastive firing analysis. This technique isolates causal features—those that consistently fire when the model performs specific tasks, such as Optical Character Recognition (OCR) or spatial reasoning. By mapping these activations, the framework identifies the precise internal "circuits" responsible for these capabilities.

3. Feature-Level Control: The final component is the ability to intervene. MMDiff allows for causal removal or steering of these feature directions. By manipulating the activation levels of specific features, developers can selectively degrade or enhance target behaviors. This is achieved by either ablating the feature (removing its influence) or steering it (amplifying its influence) during the forward pass.

This architecture was validated across three distinct MLLM families: LLaVA-MORE, PaliGemma 2, and InternVL3.5. This diversity in backbones and vision encoders demonstrates that the MMDiff pipeline is robust and generalizes well across different architectural paradigms.

Benchmark Analysis

The efficacy of MMDiff is quantified by its ability to selectively influence model performance without causing catastrophic forgetting or degrading general capabilities like Visual Question Answering (VQA). The researchers reported significant improvements in control precision and behavioral modification.

Notably, these interventions were achieved with minimal impact on general VQA performance, suggesting that the features isolated by MMDiff are highly specific to their respective tasks rather than being entangled with general model intelligence.

Developer Implications

For engineers working on production-grade MLLMs, MMDiff offers a practical toolkit for safety and alignment. Current methods for safety alignment often rely on fine-tuning or RLHF, which can be blunt instruments that degrade model utility. MMDiff provides a surgical alternative: if a model exhibits unsafe behavior related to specific visual inputs, developers can theoretically identify the responsible feature directions and suppress them directly.

Furthermore, the ability to steer features provides a new avenue for model optimization. Instead of retraining a model to improve its OCR capabilities, developers might use MMDiff to identify the features responsible for text recognition and amplify them. This could significantly reduce the compute costs associated with iterative fine-tuning. However, the framework also highlights the complexity of modern MLLMs; because these models are non-linear, the interaction between features is not always additive, meaning that aggressive steering could potentially lead to unforeseen side effects in edge cases.

Bottom Line

MMDiff represents a shift in how we approach the interpretability of multimodal systems. By moving beyond post-hoc inspection and into active, feature-level control, the framework provides a viable path toward models that are not only more transparent but also more steerable. As MLLMs become more integrated into critical infrastructure, the ability to audit and surgically adjust their behavior will likely become a standard requirement for deployment.

Pneumetron