Chimera: Scaling Hybrid Visual Diffusion Transformers with HeteroP

What Changed

The landscape of high-resolution image and long-form video generation has long been constrained by the quadratic computational complexity of standard attention mechanisms. As models scale to accommodate larger multimodal contexts, the cost of full attention becomes prohibitive, limiting the duration and fidelity of generated outputs. The introduction of Chimera marks a significant departure from these traditional architectures. By utilizing a hybrid visual diffusion backbone, Chimera replaces monolithic attention layers with a combination of specialized components designed for efficiency and long-context state tracking. This architecture not only addresses the bottleneck of quadratic scaling but also introduces a principled approach to model scaling known as HeteroP, which allows for the systematic transfer of hyperparameters across varying model widths and depths.

Technical Details

Chimera operates on a unified input stream, processing text, image, and video tokens in a single raster-ordered sequence. A defining feature of this approach is the complete removal of explicit positional embeddings, relying instead on the model's internal structural components to maintain spatial and temporal awareness. The backbone is built upon three primary pillars: Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and modality-aware short convolutions.

KDA serves as the primary engine for long-context state tracking, achieving O(N) computational complexity, which is essential for handling extended video sequences without the memory explosion associated with standard self-attention. Complementing this, interleaved MLA layers facilitate direct global interaction between tokens, ensuring that the model maintains coherence across the entire visual field. To capture local spatiotemporal nuances, the architecture employs modality-aware short convolutions, which are particularly effective at modeling the immediate relationships between adjacent frames and pixels.

Capacity is further expanded through the integration of Sparse Mixture-of-Experts (MoE) layers. By selectively activating only a subset of parameters for any given input, Chimera maintains a high total parameter count while keeping the activated compute budget strictly controlled. The scaling of this heterogeneous structure is governed by HeteroP, a module-wise scheme that calculates hyperparameter transfers based on the functional fan-in of each tensor and the overall depth of the model. This method allows for the derivation of Chinchilla-style compute-optimal laws, providing a roadmap for balancing activated model size, training-token counts, and the ratio of image-to-video data.

Benchmark Analysis

Chimera demonstrates substantial improvements in computational efficiency compared to standard full-attention baselines. In evaluations against the Wan-2.1 2B full-attention model, the Chimera dense backbone achieved 1.7x greater compute efficiency, while the complete system, leveraging the full suite of optimizations, reached 7.3x efficiency. Furthermore, the model exhibits strong zero-shot generalization capabilities. When trained on 5-second clips, Chimera successfully extrapolated to 30-second video generation with only a 6.5% degradation in Fréchet Inception Distance (FID) for the final five seconds of the output. These results suggest that the principled scaling laws derived via HeteroP are effective at maintaining model performance even when extending beyond the training distribution.

Developer Implications

For engineers and researchers working on generative visual models, Chimera provides a blueprint for moving away from monolithic transformer blocks. The success of the hybrid approach—specifically the combination of O(N) attention mechanisms and modality-specific convolutions—suggests that future architectures should prioritize specialized layers that align with the structural properties of visual data. The HeteroP scaling scheme is perhaps the most significant takeaway for practitioners, as it offers a methodology for scaling heterogeneous models that are often difficult to tune manually. By aligning model capacity with training data volume through these laws, developers can optimize their compute budgets more effectively. Furthermore, the ability to perform zero-shot temporal extrapolation without length-specific fine-tuning reduces the operational burden of training models for multiple video durations, simplifying the pipeline for production-grade video generation services.

Bottom Line

Chimera represents a shift toward more efficient, scalable, and modular visual diffusion architectures. By replacing full attention with a hybrid stack and applying rigorous scaling laws, the researchers have demonstrated that it is possible to achieve significant gains in compute efficiency while simultaneously improving long-context generation capabilities. As visual models continue to grow in complexity, the methods introduced in Chimera, particularly the HeteroP scaling framework, will likely become essential tools for balancing performance and resource constraints in large-scale AI deployment.

Pneumetron