NVIDIA Unveils Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4: A Deployment-Optimized Hybrid MoE LLM

What Changed

NVIDIA has introduced Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4, a new large language model (LLM) designed for enhanced inference efficiency. This model is a compressed variant of the Nemotron-3-Super-120B-A12B, specifically optimized for deployment in interactive, reasoning-heavy, and long-context scenarios. The core innovation lies in its use of the Iterative Puzzle post-training compression framework, which significantly reduces model size and improves throughput without substantial accuracy degradation.

The model's architecture is a hybrid MoE, incorporating Mamba, MoE, and Attention layers. A key change from its parent model is the reduction in total parameters from 120.7B to 75.3B, and active parameters from 12.8B to 9.3B. This compression is achieved through a multi-stage pipeline involving heterogeneous MoE pruning, active parameter budget reduction, and Mamba SSM state pruning. Additionally, the model supports Multi-Token Prediction (MTP) for faster text generation, a feature also present in Nemotron-3-Super.

Technical Details

Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 employs a Mamba2-Transformer Hybrid Latent Mixture of Experts (LatentMoE) architecture. The compression process, detailed in the Iterative Puzzle framework, focuses on three architectural dimensions:

1. Heterogeneous MoE Channel Pruning: The routed expert intermediate dimensions are pruned non-uniformly across MoE layers. The parent's routed expert intermediate size of 2688 is reduced to a layer-dependent range of 1280-2688, allowing for more aggressive pruning in less sensitive layers.
2. Heterogeneous Active Expert Reduction: The number of activated routed experts per token is reduced from 22 in the parent model to a layer-dependent range of 4-18. This directly impacts active parameters and boosts efficiency in compute-bound inference scenarios, such as prefill and large-batch decoding.
3. Mamba SSM State Pruning: The Mamba SSM state size is reduced from 128 to 96 channels. This optimization reduces Mamba cache I/O, improving decode-stage efficiency, particularly at larger batch sizes.

The training and optimization procedure is a multi-stage pipeline:

• Stage 1: Iterative Puzzle Compression: This involves three compression-and-recovery steps. Each step prunes the model to an intermediate target budget, followed by a short knowledge distillation recovery phase. This stage progressively reduces MoE weights and Mamba SSM state size, with knowledge distillation using 24B, 43.2B, and 52.8B tokens respectively.
• Stage 2: Long-Context Knowledge Distillation Recovery: After architectural selection, the compressed model undergoes further knowledge distillation from Nemotron-3-Super. This stage recovers quality, especially long-context capabilities, using a mixture of 30% pretraining data and 70% supervised fine-tuning data. Distillation is performed at increasing sequence lengths (32Ki, 128Ki, 512Ki) with up to 100B training tokens per phase.
• Stage 3: Reinforcement Learning (RL) Recovery: The model undergoes RL recovery, specifically targeting software-engineering and agentic capabilities, which are sensitive to compression. This stage utilizes the Nemotron-3-Super software-engineering RL pipeline (SWE-RL), including single-step tool-use comparison training and end-to-end sandbox RL.
• Stage 4: Deployment Optimization: The final checkpoint is prepared for deployment through post-training quantization. FP8 checkpoints are designed for Hopper-class GPUs, while NVFP4 checkpoints target Blackwell-class GPUs. Continued MTP training is also applied to enhance speculative decoding acceptance length and serving throughput.

The model supports a maximum context length of up to 1M tokens and is compatible with Hugging Face Transformers and vLLM runtime engines. It is optimized for NVIDIA Blackwell and Hopper hardware microarchitectures and runs on Linux.

Benchmark Analysis

Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 demonstrates performance improvements compared to its parent, Nemotron-3-Super:

• Server Throughput: Achieves approximately 2x higher server throughput on a single 8x B200 node at matched user-throughput constraints.
• H100 Concurrency: Increases sustainable 1M-token single-H100 concurrency from 1 request to 8 requests.
• Accuracy: Maintains strong accuracy across various benchmarks, including reasoning, coding, multilingual, long-context, and agentic tasks.

Specific benchmark results for the NVFP4 variant include:

• General Knowledge (MMLU-Pro): 82.2
• Reasoning: AIME25 (no tools) 89.9, HMMT Feb25 (no tools) 92.9, GPQA (no tools) 78.0, LiveCodeBench (v5 2024-07↔2024-12) 79.9, SciCode (subtask) 40.3, HLE (no tools) 15.7
• Agentic (Terminal Bench hard subset): 23.4
• TauBench V2 Average: 59.9
• Chat & Instruction Following: IFBench (prompt) 71.3, Scale AI Multi-Challenge 55.9, Arena-Hard-V2 69.0
• Long Context: AA-LCR 57.1, RULER @ 256k 95.3, RULER @ 512k 94.8, RULER @ 1M 93.2
• Multilingual: MMLU-ProX (avg over langs) 76.5, WMT24++ (en→xx) 85.1

These evaluations were conducted using NVIDIA's Nemo Evaluator SDK and Nemo Skills Harness, with specific open-source packaged containers for certain benchmarks.

Developer Implications

Developers working on AI Agent systems, chatbots, and RAG systems will find Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 a suitable choice for high-volume workloads and complex instruction-following tasks. The model's optimization for inference efficiency means developers can achieve higher throughput and concurrency on NVIDIA GPUs, particularly Blackwell-class hardware with NVFP4 checkpoints. The support for Multi-Token Prediction (MTP) further enhances text generation speed, which is critical for interactive applications.

The model's ability to handle long contexts, up to 1M tokens, makes it valuable for applications requiring extensive contextual understanding. Its multilingual support, including English, French, German, Italian, Japanese, Spanish, and Chinese, broadens its applicability for global deployments. The commercial use license (OpenMDW-1.1) also simplifies adoption for commercial projects. Integration with existing ML ecosystems is facilitated by its compatibility with Hugging Face Transformers and vLLM.

Bottom Line

NVIDIA's Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 represents a significant step in deploying efficient, high-performance LLMs. By leveraging the Iterative Puzzle compression framework, NVIDIA has successfully reduced the model's parameter count and improved inference throughput, making it highly suitable for demanding interactive and long-context AI applications. The model's hybrid MoE architecture, combined with multi-stage optimization including knowledge distillation and reinforcement learning, ensures that these efficiency gains do not come at the cost of accuracy across diverse benchmarks. For developers targeting NVIDIA's latest GPU architectures, this model offers a compelling solution for building robust and scalable AI systems.

Pneumetron