Empero AI Releases Qwythos-9B-v2: Addressing Looping and Enhancing Robustness in a 1M-Token LLM

What Changed

Empero AI has launched Qwythos-9B-v2, a significant update to its Qwythos-9B large language model. The primary objective of this release was to address and eliminate the looping and degeneration behavior observed in the previous version, particularly under greedy or low-temperature decoding. This issue, which previously affected 6.7% of greedy generations, has been reduced to 0% in v2. The update also restores the native multi-token-prediction (MTP) module, which was missing in the prior export, ensuring compatibility with speculative decoding setups.

Crucially, these improvements were implemented without compromising the model's core capabilities. Qwythos-9B-v2 retains the deep chain-of-thought reasoning, the 1M-token context window (enabled by YaRN rope-scaling), and its intentionally uncensored research posture. Another refinement includes a cleaner identity management, where the model now introduces itself only when explicitly asked, rather than prefacing unrelated answers with its identity.

Technical Details

The core of the looping fix in Qwythos-9B-v2 lies in a technique called Final-Token Preference Optimization (FTPO). This method identifies the specific token that initiates a repetition loop and then gently trains the model to favor coherent alternatives at that precise position. This targeted approach ensures that the rest of the model's knowledge and reasoning capabilities remain unaffected.

The training procedure involved fine-tuning the base Qwythos (Qwythos-9B-Claude-Mythos-5-1M) using approximately 2,000 preference tuples. These tuples were automatically mined by eliciting looping behavior at low temperatures and extracting the rejected loop token versus the model's own coherent top-k alternatives at each loop-start position. The fine-tuning utilized LoRA with parameters r=256, α=128, lr=1.5e-5 for one epoch, with early stopping based on chosen_win ≥ 0.30. This 'light touch' training targeted all attention, MLP projections, and the lm_head.

The model is built on the Qwen3.5-9B architecture, featuring a hybrid design (3:1 Gated-DeltaNet linear-attention : full attention) and is multimodal-capable, though its practical usage is text-only. It utilizes bfloat16 parameters and safetensors. The 1,048,576-token context is achieved through YaRN rope-scaling with a factor of 4, extending the native 262,144-token window. The tokenizer and chat template are Qwen3.5 native (ChatML-style).

Benchmark Analysis

Empero AI conducted internal evaluations to confirm that the hygiene upgrades did not lead to a capability regression. The benchmarks were measured using an internal harness with generative chain-of-thought, greedy/pass@1 decoding, and an independent LLM grader for quality metrics. Sample sizes included MMLU/ARC/GSM8K n=500, GPQA-diamond n=198, and HumanEval n=164.

As the table indicates, Qwythos-9B-v2 maintains or slightly improves upon the reasoning and knowledge benchmarks compared to its predecessor and the base Qwen3.5-9B model. For instance, MMLU (CoT) remains at 83.8%, ARC-Challenge improved to 96.4%, and GSM8K reached 93.6%. The most significant improvement is the elimination of the greedy looping rate, which dropped from 6.7% in Qwythos-9B to 0.0% in v2.

It is noted that HumanEval (pass@1) scores for v2 are 77.4%, a slight decrease compared to the raw Qwen3.5-9B base (81.7%) and Qwythos-9B (79.9%). This is acknowledged as a small, known cost associated with the reasoning/looping-fix fine-tuning. The model's MMLU score is notably higher with Chain-of-Thought (CoT) reasoning (83.8%) compared to a 5-shot loglikelihood setup (69.6%), underscoring its strength as a reasoning model.

Developer Implications

For developers, Qwythos-9B-v2 offers a more robust and reliable text generation experience. The elimination of looping behavior means that repetition_penalty is no longer a critical parameter for maintaining coherence, allowing for more straightforward deployment and potentially better performance with greedy or low-temperature decoding. This reduces the need for heuristic tuning of generation parameters.

The restoration of the native MTP head ensures that the model's configuration and weights are in agreement, which is beneficial for developers implementing speculative decoding. While the MTP head was not co-trained with the fine-tuned weights, its presence allows for better integration into advanced decoding pipelines.

With a 1M-token context, Qwythos-9B-v2 is well-suited for applications requiring extensive context understanding, such as long-form content generation, complex document analysis, and detailed code review. The model's uncensored nature makes it particularly valuable for specialized research, cybersecurity, red-teaming, and scientific domains like biology, chemistry, pharmacology, and clinical work, where unconstrained inquiry is often necessary. Developers must, however, deploy it responsibly and in compliance with applicable laws.

Usage with the transformers library is straightforward, as demonstrated by the provided Python snippet. For serving, vLLM is supported out-of-the-box, with a recommendation to use --limit-mm-per-prompt '{"image":0,"video":0}' to streamline startup given its text-only practical application.

Bottom Line

Qwythos-9B-v2 represents a focused and impactful update from Empero AI. By leveraging Final-Token Preference Optimization, the model effectively resolves critical looping issues without sacrificing its strong reasoning capabilities or its extensive 1M-token context. While there's a minor trade-off in HumanEval scores, the overall enhancement in generation robustness and the restoration of the MTP head make Qwythos-9B-v2 a more stable and developer-friendly option for applications demanding deep reasoning, long context, and an uncensored research stance. This release solidifies its position as a valuable tool for technical and scientific domains.

Pneumetron