TurboQuant: A Rust Vector Index Outperforming FAISS in Memory and Speed

What Changed

RyanCodrai has released turbovec, a new vector index built in Rust with Python bindings, based on Google Research's TurboQuant algorithm. This development introduces a data-oblivious quantizer designed for near-optimal distortion without requiring a separate training phase. turbovec addresses critical challenges in vector search, particularly memory consumption and search latency, by offering substantial compression and optimized search kernels. A notable improvement is its ability to handle a 10 million document corpus using only 4 GB of RAM, a significant reduction from the 31 GB typically required for float32 representations, while simultaneously delivering faster search performance than FAISS.

Key features include online ingest capabilities, allowing vectors to be indexed as they are added without a training step or rebuilds. It also provides fast SIMD search through hand-written NEON (ARM) and AVX-512BW (x86) kernels. The index supports filtering at search time, enabling developers to restrict results to a specified allowlist without over-fetching or incurring recall penalties. Furthermore, turbovec is designed for purely local operation, making it suitable for privacy-sensitive RAG stacks.

Technical Details

turbovec implements Google's TurboQuant algorithm, which operates by compressing high-dimensional vectors. The process begins by normalizing each vector to a unit direction on a hypersphere, storing its original length as a single float. Subsequently, all vectors undergo a random orthogonal rotation. This rotation is crucial as it ensures that each coordinate independently follows a predictable Beta distribution, converging to a Gaussian N(0, 1/d) in high dimensions, regardless of the input data.

For improved accuracy, turbovec incorporates a per-coordinate calibration step (TQ+). This fits two scalars (shift and scale) per coordinate during the initial vector addition, mapping empirical 5/95% quantiles to the canonical Beta marginal. This calibration is frozen after the first add, ensuring no retraining is needed for subsequent additions. Following calibration, Lloyd-Max scalar quantization is applied, precomputing optimal bucket boundaries and centroids based on the known distribution. This quantizes each coordinate into a small integer (e.g., 0-3 for 2-bit, 0-15 for 4-bit), which are then tightly bit-packed. For instance, a 1536-dimensional FP32 vector, originally 6,144 bytes, is compressed to 384 bytes at 2-bit, achieving 16x compression.

To mitigate the systematic underestimation of inner products caused by scalar quantization, turbovec employs length-renormalized scoring. It computes and stores a scalar ||v|| / ⟨u, x̂⟩ for each compressed vector, where u is the rotated unit vector and x̂ is its centroid reconstruction. During search, this scalar is multiplied by the per-candidate score, correcting the inner-product estimator without additional search-time cost or storage. Search operations involve rotating the query once and scoring directly against codebook values using SIMD intrinsics (NEON on ARM, AVX-512BW on x86 with AVX2 fallback) and nibble-split lookup tables.

Benchmark Analysis

turbovec demonstrates significant performance advantages over FAISS IndexPQFastScan in several benchmarks, typically conducted on 100K vectors, 1K queries, and k=64, with results representing the median of 5 runs.

Compression: A 10 million document corpus, which would occupy 31 GB of RAM as float32, is reduced to 4 GB using turbovec.

Recall: Against FAISS IndexPQ (LUT256, nbits=8), turbovec shows competitive to superior recall. On OpenAI embeddings (d=1536 and d=3072), TurboQuant beats FAISS by 0.2–1.9 points at R@1 across 2-bit and 4-bit configurations, with both reaching 1.0 by k=8 (≥0.997 at k=4). For GloVe d=200, TurboQuant beats FAISS by 0.9 points at 4-bit and is effectively tied at 2-bit (within 0.1 points) at R@1.

Search Speed:

• ARM (Apple M3 Max): turbovec consistently outperforms FAISS FastScan by 10–19% across all configurations in both single-threaded and multi-threaded scenarios.
• x86 (Intel Xeon Platinum 8481C / Sapphire Rapids, 8 vCPUs): turbovec wins the 4-bit configurations by up to approximately 5%. In multi-threaded d=3072, performance is tied. For 2-bit configurations, turbovec is modestly behind FAISS, with the most noticeable difference being about 8% on d=1536 single-threaded, where FAISS's AVX-512 VBMI path has an edge.

Developer Implications

Developers working with vector search and RAG architectures now have a compelling alternative to existing solutions like FAISS, particularly when memory constraints, privacy, or latency are critical. The substantial memory compression offered by turbovec means larger datasets can be managed on less expensive hardware, or more data can be held in memory for faster access. Its online ingest capability simplifies index management, eliminating the need for retraining or rebuilding as the corpus evolves.

The Python bindings make turbovec accessible to a broad developer base, with pip install turbovec enabling quick integration. For applications requiring stable IDs that persist through deletions, IdMapIndex provides O(1) removal by ID. The direct filtering at search time is a significant advantage for hybrid retrieval systems, allowing developers to efficiently narrow down results from external systems (e.g., SQL, BM25) before dense re-ranking, avoiding unnecessary computation.

turbovec also offers drop-in replacements for in-tree reference vector/document stores in popular frameworks like LangChain, LlamaIndex, Haystack, and Agno. This allows developers to swap out existing in-memory stores with turbovec by changing a single import, maintaining the same public API and persistence semantics, thus minimizing integration effort.

Bottom Line

turbovec represents a notable advancement in vector indexing technology, offering a robust, memory-efficient, and performant solution built on the TurboQuant algorithm. Its ability to drastically reduce memory footprint while maintaining or improving search speeds, coupled with features like online ingest, efficient filtering, and local operation, positions it as a strong contender for developers building RAG systems where resource optimization and data privacy are paramount. The availability of Python and Rust interfaces, along with direct framework integrations, lowers the barrier to adoption for a wide range of AI/ML applications.

Pneumetron