Bonsai-27B: A 1-Bit LLM for On-Device Inference with Llama.cpp and MLX

What Changed

Prism ML has introduced Bonsai-27B, a 27B-class language model that leverages binary transformer weights, achieving a deployed footprint of approximately 3.9 GB. This represents a significant reduction in size, roughly 14.2 times smaller than its FP16 counterpart, making it feasible for execution on consumer-grade hardware such as laptops and single GPUs. The model is distributed in the GGUF Q1_0_g128 format, specifically designed for llama.cpp, and also offers an MLX companion for native Apple Silicon inference, including mobile devices like the iPhone.

Bonsai-27B distinguishes itself by retaining a substantial portion of its reasoning and agentic behavior, with an average of 76.11 across 15 thinking-mode benchmarks, which is 89.5% of the FP16 reference. This performance is notable given its true 1.125 bits per weight, a level where conventional low-bit representations typically experience a collapse in intelligence. A key feature is its end-to-end binary language weights across embeddings, attention projections, MLP projections, and the LM head, without relying on higher-precision escape hatches often found in other low-bit models.

Additionally, Bonsai-27B supports an extended 262K-token context on-device, facilitated by its Qwen3.6-27B hybrid-attention backbone (approximately 75% linear attention) and 4-bit KV-cache quantization. The model also ships with a DSpark speculative-decoding drafter layer, which provides a lossless 1.37x decode speedup on the CUDA serving path.

Prism ML also offers a quality-oriented operating point, Ternary Bonsai 27B, which uses 1.71 bits per weight, results in a 5.9 GB footprint, and retains 94.6% of FP16 intelligence.

Technical Details

Bonsai-27B is derived from Qwen3.6-27B, maintaining its 27B hybrid-attention causal language model architecture. It comprises approximately 27.3 billion binary language weights (24.8B backbone across 64 blocks + 2.5B embedding/LM head) and an optional 0.46 billion vision tower. The architecture incorporates hybrid attention (approximately 75% linear / 25% full attention), SwiGLU MLP, RoPE, and RMSNorm.

The core innovation lies in its GGUF Q1_0_g128 weight format, where each weight is a single sign bit (0 maps to -scale, 1 maps to +scale), and every group of 128 weights shares one FP16 scale factor. This yields an effective 1.125 bits per weight, achieving the stated 14.2x reduction versus FP16. This aggressive quantization applies to embeddings, attention projections, MLP projections, and the LM head. The vision tower, if used, ships in HQQ 4-bit (Q8_0 container) at 0.63 GB.

Memory management is optimized for on-device inference. The 4-bit KV cache, which is near-lossless, helps manage the context length. For example, a 100K-token context can be held at 11.6 GB without KV-cache compression, fitting mainstream laptops. With the 4-bit KV cache enabled, the 100K peak drops to approximately 6.8 GB, and the full 262K window fits within approximately 9.4 GB peak memory.

Bonsai-27B integrates custom 1-bit hybrid-attention kernels for llama.cpp (CUDA, Metal) and is also available for MLX for Apple Silicon. The DSpark speculative-decoding drafter layer, a compact six-layer block-parallel transformer, is trained against the Bonsai 27B target. It adds roughly 0.5 GB at serving precision and is shipped 4-bit quantized (Q4_1 pack) by default. This drafter provides a lossless speedup on the CUDA serving path.

Benchmark Analysis

Evaluations were conducted using EvalScope + vLLM on NVIDIA H100 in