Prism ML's Ternary Bonsai 27B: 2-bit LLM for On-Device Inference

What Changed

Prism ML has introduced Ternary Bonsai 27B, a 27-billion-parameter class language model that leverages a ternary quantization scheme to achieve a significantly reduced footprint while retaining high reasoning capabilities. This model is derived from Qwen3.6-27B and is specifically optimized for on-device deployment on platforms like Apple Silicon and NVIDIA GPUs. The core innovation lies in its ternary g128 weight representation, where each weight takes a value from {−1, 0, +1}, combined with group-wise FP16 scaling. This approach results in an effective storage cost of approximately 1.71 bits per weight, leading to an idealized 9.4x reduction compared to FP16.

Key features of this release include a deployed footprint of around 7.2 GB, enabling the model to run on everyday laptops and single GPUs. Despite this aggressive quantization, Ternary Bonsai 27B retains 95% of the intelligence of its FP16 counterpart, as measured by an 80.49 average across 15 thinking-mode benchmarks. It also maintains complex capabilities such as math, coding, and agentic tool use, areas where conventional low-bit representations often degrade significantly. The model incorporates end-to-end ternary language weights across all major components, including embeddings, attention projections, MLP projections, and the LM head, without relying on higher-precision escape hatches. It also supports a 262K-token context length on-device, facilitated by a Qwen3.6-27B hybrid-attention backbone (approximately 75% linear attention) and 4-bit KV-cache quantization. Custom 2-bit hybrid-attention kernels are provided for Apple MLX (Python, Swift) and CUDA, consuming packed weights directly without expansion to FP16.

Additionally, Ternary Bonsai 27B ships with a DSpark speculative-decoding drafter layer, trained against the Bonsai 27B target. This drafter provides a lossless 1.34x decode speedup on the CUDA serving path. A 1-bit companion model, 1-bit Bonsai 27B, is also available for phone-class devices, fitting within the memory constraints of devices like the iPhone 17 Pro Max.

Technical Details

Ternary Bonsai 27B is based on the Qwen3.6-27B architecture, a 27B hybrid-attention causal language model. It comprises approximately 27.3 billion ternary language weights, with an optional 0.46 billion parameter vision tower. The architecture features hybrid attention (approximately 75% linear / 25% full attention), SwiGLU MLP, RoPE, and RMSNorm. The model supports a 262K-token context length, made practical by its predominantly linear-attention backbone and near-lossless 4-bit KV quantization. The KV cache for the hybrid backbone grows a full-attention cache on only 16 of 64 layers, resulting in approximately 4.3 GB at the full 262K window.

The core of the quantization is the Ternary g128 weight format, where each weight is one of {−1, 0, +1}, with a shared FP16 scale factor for every group of 128 weights. This results in an effective storage cost of approximately 1.71 bits per weight. The deployed footprint for the language model alone is about 7.2 GB. The model's low-bit coverage extends to embeddings, attention projections, MLP projections, and the LM head. The vision tower, if used, is in HQQ 4-bit format and is optionally loaded only for image input.

For acceleration, a DSpark speculative-decoding drafter layer is provided. This drafter is a compact six-layer block-parallel transformer conditioned on hidden states from the target model. It adds approximately 0.5 GB to the serving precision and is shipped 4-bit quantized (Q4_1 pack) by default. The drafter's precision affects only speed, not output quality, due to the lossless nature of speculative decoding. Backends supported include Apple MLX (Python, Swift) and CUDA, with custom low-bit kernels for both platforms.

Regarding memory requirements, while the ideal size for Ternary g128 is 5.9 GB, current kernels store each ternary value in a 2-bit slot, leading to a deployed footprint of approximately 7.2 GB. This figure describes the language model only, as normalization and scale parameters remain in higher precision. The MLX packaging, which stores both a scale and a bias per group, results in an on-disk size of 8.49 GB, with an effective rate of 2.25 bits/weight due to current MLX limitations regarding scale-only group formats.

Benchmark Analysis

Performance benchmarks were conducted using EvalScope + vLLM on NVIDIA H100, focusing on

Pneumetron