HOMIE: Advancing Human-Object Centric Video Personalization

What Changed\n\nSubject-driven video generation has evolved rapidly, yet the specific domain of Human-Object Centric Video Personalization (HOCVP) remains a significant challenge for generative models. Current state-of-the-art approaches often struggle to maintain a balance between two competing objectives: high subject fidelity and the accurate representation of interaction patterns between humans and objects. This is particularly evident when the objects involved are abstract, such as corporate logos or complex, non-standard shapes. Furthermore, while developers have attempted to use intra-subject references—such as OCR maps or multi-view inputs—to guide generation, most existing architectures lack the necessary mechanisms to interpret the latent correspondence between these references and the target video frames.\n\nHOMIE (Human-object Centric Video Personalization via Multimodal Intelligent Enhancement) represents a shift in how these models handle multimodal inputs. Rather than relying on standard, often costly re-alignment techniques that can diminish the controllability of text encoders, HOMIE introduces a unified framework. It effectively bridges the gap between inter-subject personalization (where a specific subject is placed in a new context) and intra-subject personalization (where specific reference materials are used to guide the generation of a known subject). By integrating Multimodal Large Language Models (MLLMs) directly into the generation pipeline, the framework extracts deeper semantic relationships without sacrificing the granular control required for high-quality video synthesis.\n\n## Technical Details\n\nThe architecture of HOMIE is built upon three primary innovations designed to improve the alignment of semantic and visual features. The first is a strategic MLLM integration. Unlike previous methods that often struggle with the overhead of re-aligning text encoders, HOMIE utilizes the MLLM to extract knowledge of reference-level relationships. This allows the model to understand the context of the interaction—such as how a human hand should naturally grasp a specific object—before the diffusion process begins.\n\nTo ensure these semantic insights are effectively utilized, the researchers introduced global multimodal guidance within the self-attention layers of the model. This mechanism acts as a bridge, aligning the semantic features derived from the MLLM with the VAE (Variational Autoencoder) tokens. By injecting this guidance directly into the self-attention mechanism, the model can maintain spatial consistency across frames, ensuring that the object remains coherent throughout the duration of the video. This is a significant departure from standard cross-attention mechanisms, which often fail to capture the long-range dependencies required for complex human-object interactions.\n\nFinally, the framework employs a modality-reference embedding layer. This component is designed to differentiate between tokens derived from MLLM features and those originating from VAE tokens. By assigning distinct embeddings to these different data types, the model can explicitly associate intra-subject reference image tokens with the corresponding generation targets. This differentiation prevents the 'token bleeding' that often occurs when multiple sources of information are fed into a transformer-based architecture, resulting in cleaner, more accurate video outputs.\n\n## Developer Implications\n\nFor engineers and researchers working in the generative video space, HOMIE offers several practical advantages. First, the unified approach to inter- and intra-subject personalization simplifies the training pipeline. Instead of maintaining separate models for different types of personalization tasks, developers can utilize a single framework that adapts to the input modality, whether it be a simple text prompt, a multi-view reference, or an OCR-based map.\n\nThe integration of MLLM-based guidance also provides a more robust way to handle abstract objects. For developers working on brand-centric video generation—where the accurate depiction of logos or proprietary products is non-negotiable—the ability to leverage MLLM semantic knowledge to guide the model is a game-changer. It reduces the reliance on extensive fine-tuning for every new object, as the model can better generalize from reference images.\n\nFurthermore, the modality-reference embedding strategy provides a cleaner interface for incorporating auxiliary inputs. If a project requires the use of depth maps, pose estimation, or multi-view references, the HOMIE architecture provides a structured way to inject this data without disrupting the core text-to-video generation process. This modularity is essential for building production-grade video generation systems that require high levels of controllability and consistency.\n\n## Bottom Line\n\nHOMIE addresses the fundamental limitations of existing HOCVP methods by providing a unified, MLLM-enhanced framework for video personalization. By focusing on the latent correspondence between reference inputs and generated frames, it achieves a superior balance between subject fidelity and interaction accuracy. As the field of generative video continues to mature, frameworks like HOMIE, which prioritize intelligent multimodal integration over brute-force re-alignment, will likely become the standard for high-fidelity, controllable video synthesis.