Skip to content

📝 Keyword Retrieval (BM25)

Spector integrates a high-performance BM25 keyword matching layer directly into its memory retrieval pipeline. This provides exact term-matching capabilities (such as search for specific error codes, variable names, UUIDs, or project references) that dense semantic vectors can sometimes miss.

🧮 BM25 Scoring Formula

Spector uses the standard Okapi BM25 scoring algorithm. Given a query \(Q\) containing terms \(q_1, q_2, \dots, q_n\), the BM25 score of a document \(D\) is:

\[Score(D, Q) = \sum_{i=1}^{n} IDF(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{avgdl}\right)}\]

Where: - \(f(q_i, D)\) is the term frequency of term \(q_i\) in document \(D\). - \(|D|\) is the document length (in words/tokens). - \(avgdl\) is the average document length across the entire index. - \(k_1\) is a parameter controlling term frequency saturation (defaults to \(1.2\)). - \(b\) is a parameter controlling document length normalization (defaults to \(0.75\)).

The Inverse Document Frequency (\(IDF\)) is calculated as:

\[IDF(q_i) = \ln \left( \frac{N - n(q_i) + 0.5}{n(q_i) + 0.5} + 1 \right)\]

Where \(N\) is the total number of documents in the index, and \(n(q_i)\) is the number of documents containing term \(q_i\).

⚡ Hardware-Level SIMD Acceleration

Traditional search engines compute term frequencies and BM25 scores sequentially or via complex pointer traversal, leading to significant CPU cache misses. Spector solves this by organizing indices into cache-friendly layouts and leveraging hardware SIMD instruction sets:

  • Contiguous Memory Layout: Document lengths, term frequencies, and postings tables are stored in continuous memory blocks to maximize L1/L2 cache hit rates.
  • Vectorized Score Accumulation: Scoring loops are compiled into native SIMD instructions, processing multiple documents (e.g., 8 documents on 256-bit AVX2, or 16 documents on 512-bit AVX-512) in parallel in a single CPU clock cycle.
  • Thread Safety: Inverted indexes are accessed via lock-free memory structures, yielding zero garbage collection overhead during concurrent retrieval sweeps.

⚙️ Configuration & Execution

BM25 retrieval can be enabled globally or toggled at the query level.

Global defaults in spector.yml

spector:
  memory:
    bm25-enabled: true     # Active by default

Query-level settings

Configure options using query parameters:

  • gamma: Scaling weight applied to the BM25 score when merged.
  • enableTextSearch: Set to false to completely disable the BM25 lexical check for a specific query.
  • textSearchMode: Set to KEYWORD_ONLY to execute only the BM25 lexical search path, bypassing dense HNSW calculations entirely.

🛑 Limitations of Exact Matching

While BM25 is highly effective for exact terms, it suffers from two key drawbacks: 1. Vocabulary Mismatch: If a user searches for "automobile" but the document only contains "car", BM25 will yield a score of 0.0. 2. Lack of Semantic Context: It does not capture sentence structure or underlying concepts.

To bridge this gap, Spector's memory recall pipeline combines BM25 with SPLADE Learned Sparse Retrieval (which expands documents with neural synonyms) and Dense Vector Similarity, fusing their scores using RRF before performing final graph expansions.