Prism ML Introduces Bonsai 27B: A 1-bit LLM for On-Device Inference

What Changed

Prism ML has unveiled Bonsai 27B, a significant advancement in on-device large language models (LLMs). This model is notable for its use of 1-bit binary transformer weights, making it the first 27B-class model capable of running on a smartphone. The core innovation lies in its highly compressed architecture, which reduces the deployed footprint to approximately 3.9 GB, a 14.2x reduction compared to its FP16 counterpart (54 GB).

Bonsai 27B retains substantial reasoning and agentic behavior, even in the sub-4-bit regime where many conventional low-bit representations experience significant performance degradation. It achieves this through end-to-end binary language weights across embeddings, attention projections, MLP projections, and the LM head, operating at a true 1.125 bits per weight. This contrasts with other 'low-bit' models that often employ higher-precision 'escape hatches,' resulting in a higher true average bit-width than advertised.

Another key feature is its support for a 262K-token context on-device, facilitated by a Qwen3.6-27B hybrid-attention backbone (approximately 75% linear attention) and 4-bit KV-cache quantization. The model also ships with a DSpark speculative-decoding drafter layer, specifically trained against the Bonsai 27B target, which offers a lossless 1.37x decode speedup on the CUDA serving path.

Technical Details

Bonsai 27B is derived from Qwen3.6-27B, maintaining its 27.3 billion binary language weights (24.8B backbone across 64 blocks + 2.5B embedding/LM head) and an optional 0.46B vision tower. Its architecture incorporates hybrid attention (75% linear / 25% full attention), SwiGLU MLP, RoPE, and RMSNorm. The model's extensive 262K token context length is made practical on-device by its predominantly linear-attention backbone and a near-lossless 4-bit KV cache quantization, which only grows a full-attention cache on 16 of 64 layers.

The unique weight representation, termed 'Binary g128,' assigns each weight a single sign bit (0 maps to -scale, 1 maps to +scale). Every group of 128 weights shares one FP16 scale factor, resulting in an effective 1.125 bits per weight. This aggressive quantization minimizes both storage footprint and weight traffic during decoding, enabling the model to fit within a phone's memory budget.

Prism ML has developed custom 1-bit hybrid-attention kernels for Apple MLX (Python, Swift) and CUDA. These kernels consume packed weights directly, avoiding expansion back to FP16, which is crucial for efficiency. The DSpark speculative-decoding drafter, a six-layer block-parallel transformer, is conditioned on hidden states from five evenly spaced layers of the target model. It adds approximately 0.5 GB at serving precision and is shipped as a 4-bit quantized pack (1.79 GB Q4_1 by default), which performs faster than its bf16 reference while preserving output quality due to lossless verification.

The model's memory requirements demonstrate its efficiency. While the FP16 baseline is 54 GB and a conventional '4-bit' Qwen3.6-27B is 17.6 GB (true 5.2 bits/weight), Bonsai 27B's binary g128 format occupies 3.9 GB. This allows it to hold a 100K-token context at 11.6–12.2 GB without KV-cache compression, fitting mainstream laptops. With 4-bit KV cache enabled, the 100K peak drops to approximately 6.8 GB, and the full 262K window fits in about 9.4 GB peak memory.

Benchmark Analysis

Bonsai 27B was evaluated using EvalScope + vLLM on NVIDIA H100 in 'thinking mode,' designed to stress the model's reasoning capabilities across 15 benchmarks in six skill categories. The benchmarks highlight the model's ability to retain intelligence in the sub-4-bit regime, where conventional low-bit methods often collapse.

Bonsai 27B achieved an average score of 76.11 across the thinking-mode benchmarks, retaining 89.5% of the FP16 reference intelligence. This performance is notable given its extreme quantization, outperforming conventional '2-bit' builds like Qwen3.6-27B IQ2_XXS (85.5% of FP16) and Gemma-4-31B Q2_K_XL (86.2% of FP16) in terms of relative intelligence retention at a significantly lower bit-width and footprint. The degradation in conventional low-bit builds is concentrated on benchmarks requiring sustained reasoning chains.

Cross-platform throughput measurements further illustrate the model's efficiency:

¹ On the iPhone 17 Pro Max, decode is thermally limited, with sustained generation settling slightly below the cold peak. A continuous battery-drain run sustained 10.8 tok/s and generated 672 tokens per 1% of battery. The DSpark drafter provides a 1.37x end-to-end decode speedup on H100 (104.8 → 143.8 tok/s) with an accepted length of approximately 3.6 at draft depth k=4.

Energy efficiency is also a highlight, with decode energy on the M5 Pro measuring 0.275 mWh/token (with DSpark enabled), which is an order of magnitude more energy-efficient per token than datacenter GPUs (0.63–1.32 mWh/token).

Developer Implications

The release of Bonsai 27B opens new avenues for developers aiming to integrate powerful LLMs into edge devices. The model's minimal 3.9 GB deployed footprint allows it to fit within the per-app memory budget of high-end smartphones, enabling true interactive 27B-class generation on mobile platforms. This significantly lowers the barrier for deploying complex AI functionalities directly on user devices, enhancing privacy, reducing latency, and decreasing reliance on cloud infrastructure.

Developers can leverage the provided MLX (Apple Silicon) and CUDA backends, with custom 1-bit hybrid-attention kernels, for efficient inference. The availability of a DSpark speculative-decoding drafter layer further optimizes performance on CUDA, offering a substantial speedup without compromising output quality. For Apple Silicon, the drafter is not enabled by default for batch-1 verification due to current amortization limitations.

The extensive context length of 262K tokens, combined with efficient KV-cache quantization, means developers can build applications that process and generate long-form content directly on-device, such as summarizing lengthy documents or engaging in extended conversational AI. The Apache 2.0 license facilitates broad adoption and integration into various projects.

Bottom Line

Prism ML's Bonsai 27B represents a significant leap in efficient LLM deployment, demonstrating that 27B-class reasoning can be achieved on edge devices with minimal memory footprint and impressive performance. By utilizing true 1-bit binary weights and custom kernels, the model overcomes the limitations of conventional low-bit quantization, retaining high intelligence while dramatically reducing resource requirements. This innovation paves the way for a new generation of private, low-latency, and energy-efficient AI applications running directly on consumer hardware, from smartphones to laptops. The ability to run a model of this scale and capability on a phone marks a pivotal moment for the democratization of advanced AI.

Pneumetron