Wan-Dancer-14B: A Hierarchical Approach to Minute-Scale Music-to-Dance Video Generation

What Changed

Wan-AI has released Wan-Dancer-14B, a novel image-to-video generation model specifically designed for music-to-dance synthesis. This model distinguishes itself by employing a hierarchical framework that addresses the challenge of generating long-duration, coherent dance videos. Unlike single-stage generation approaches, Wan-Dancer-14B separates the process into two distinct phases: global keyframe planning and local temporal refinement. This allows the model to maintain both overall structural consistency and fine-grained rhythmic accuracy over extended video sequences.

The release includes the model weights and inference code, making it accessible for developers to experiment with and integrate into their projects. The model supports various dance genres, including Chinese Classical Dance, K-Pop Dance, Street Dance, Latin Dance, and Tap Dance, by utilizing specific prompt files for each style.

Technical Details

Wan-Dancer-14B operates on a two-stage hierarchical framework. The core idea is to first establish the global structure and then refine the local temporal details, leveraging the full-track musical context to ensure long-range coherence in the generated dance.

Stage 1: Global Keyframe Video Generation This initial stage focuses on planning the overarching movement and structure of the dance. It takes as input a reference image, a music file, and a prompt file specifying the desired dance style. Key parameters for this stage include:

• seed: For reproducibility of results.
• image_path: A path to a reference image that defines the dancer's appearance.
• prompt_path: Specifies the dance style using predefined text prompts (e.g., 古典舞_global.txt for Chinese Classical Dance).
• music_path: The audio input that dictates the rhythm and tempo of the dance.
• output_folder: Directory for saving the generated global video.
• timestamp: An identifier for output files.
• num_inference_steps: The number of diffusion inference steps, with higher values (e.g., 48) recommended for longer videos.
• cfg_scale: Classifier-free guidance scale, typically set to 5 in examples.

The output of this stage is a global keyframe video, which provides the foundational movements and overall flow of the dance, ensuring alignment with the music's global structure.

Stage 2: Final High-Resolution Video Generation (Local Refinement) Following the global planning, the second stage refines the generated video to produce a high-resolution output with enhanced temporal continuity and detail. This stage takes the global video from Stage 1 as a crucial input, along with the original reference image, music, and a local prompt file for the dance style (e.g., 古典舞_local.txt).

Additional required parameters for this stage include:

• global_video_path: The path to the video generated in Stage 1.
• prompt_path: A style-specific prompt file tailored for local refinement.

Other parameters like seed, image_path, music_path, num_inference_steps, and cfg_scale are used similarly to Stage 1. The num_inference_steps in this stage might be adjusted (e.g., 24 in examples) depending on the desired level of detail and computational resources.

The model's architecture leverages components from existing open-source projects like DiffSynth-Studio and Wan2.1, indicating an iterative development approach building on established diffusion model techniques. The installation process involves cloning the repository, setting up a Python virtual environment, and installing specific versions of dependencies such as torch, torchvision, diffusers, flash_attn, xfuser, and transformers.

Developer Implications

For developers, Wan-Dancer-14B offers a robust framework for generating expressive and synchronized dance videos. The clear two-stage process provides modularity, allowing for potential customization or optimization of each stage independently. The availability of model weights and inference code on Hugging Face simplifies deployment and experimentation.

The ability to control dance style through prompt files and incorporate a reference image opens avenues for creating personalized or character-specific dance content. This could be particularly useful in animation, virtual reality, gaming, and content creation industries where generating realistic and diverse character movements is critical.

Developers can integrate Wan-Dancer into their workflows by following the provided installation and execution scripts. The huggingface-cli and modelscope-cli commands facilitate easy model download. The explicit parameterization for seed, num_inference_steps, and cfg_scale offers granular control over the generation process, enabling fine-tuning for specific output requirements or computational constraints. The project's TODO list also indicates future plans for ComfyUI integration, which would further enhance its accessibility and ease of use for a broader developer community.

Bottom Line

Wan-Dancer-14B represents a significant advancement in music-to-dance video generation, offering a hierarchical framework that produces minute-scale, coherent, and high-quality dance sequences. By decoupling global structure planning from local temporal refinement, the model effectively addresses challenges related to long-duration consistency. Its release on Hugging Face, complete with inference code and support for diverse dance styles, provides developers with a powerful tool for creating dynamic and rhythmically accurate dance content across various applications. The project's open-source nature and future integration plans suggest a growing ecosystem for advanced video generation capabilities.

Pneumetron