How to Combine BM25 and Vector Search in One Query for RAG

Every developer building retrieval augmented generation eventually watches pure semantic search fail on a basic identifier. A user pastes an exact error string like ERR_AUTH_0x4F, a part number, or a colleague's surname, and your dense embedding model returns four chunks about generic authentication concepts. The exact token was sitting in your database. Its semantic proximity just got washed out in vector space.

Most teams fix this with a hybrid retrieval pipeline. You keep dense embeddings in a vector database, stand up an inverted index like Elasticsearch, OpenSearch, or Postgres FTS for BM25, and run reciprocal rank fusion in application code. That pattern works. It also means two separate data stores, custom synchronization workers, and retrieval drift. This guide shows how to combine BM25 and vector search properly: first with the standard two-system DIY approach, then by putting both indexes inside a single engine, HelixDB.

Step 1: Find the queries where pure vector search fails (exact ids, error codes, names)

Pure vector search maps natural language queries into a high-dimensional continuous space. Dense encoders know that 'headache medication' relates to 'ibuprofen', but they compress out exact token boundaries. When a user enters a string with zero conceptual variance, pure vector search routinely returns the wrong documents.

Build an evaluation set around three categories where dense retrieval consistently collapses. First, exact system identifiers: SKU numbers like SKU-99120-X, UUIDs, git commit hashes, and model identifiers. Embedding models treat arbitrary alphanumeric strings as subword fragments or out-of-vocabulary noise, so their vectors smear across broad semantic areas.

Second, exact error codes and stack traces: ERR_CONN_RESET, status 502, or database exception signatures. A user searching for a status code wants the exact line containing that string, not a high-level explanation of network topology. Third, low-frequency proper nouns: customer names, domain-specific acronyms, and internal team handles. Unless those tokens showed up thousands of times during embedding pretraining, the model assigns them generic vector weights.

Take twenty real queries from your production logs that match these categories. Run them through your existing vector retriever with k=10 and measure Recall@10 on exact matches. Hit rates often drop below 40% on alphanumeric strings. That gap is the reason to combine BM25 and vector search instead of relying on cosine similarity alone.

Step 2: Run BM25 and vector retrieval side by side and keep both ranked lists

The standard DIY hybrid architecture runs two concurrent queries from your application layer. When a user sends a query string, your backend runs a text search against an inverted index such as Postgres full-text search, OpenSearch, or BM25, and at the same time calls an embedding API and queries your vector database.

Don't try to merge raw scores during retrieval. BM25 produces unbounded positive floating-point values based on term frequency, inverse document frequency, and document length normalization. Depending on query length and corpus statistics, a top BM25 score might be 4.2 on one query and 28.6 on another. Vector search score ranges vary by metric and implementation: cosine similarity conventionally ranges from -1.0 to 1.0, whereas Euclidean distance is nonnegative and unbounded above. Comparing a BM25 score of 12.4 against a cosine similarity of 0.82 is mathematically meaningless.

Keep two distinct, sorted lists of document identifiers with their integer ranks instead. Your vector query returns an ordered list where the closest document has rank 1, the next has rank 2, up to rank k. Your BM25 query returns a separate ordered list from rank 1 to rank k. You can inspect the root cause of retrieval mismatches in why hybrid search returns fewer results than you asked for. In Node.js or Python, issue both requests concurrently with Promise.all or asyncio.gather so the round trips don't add up.

Step 3: Fuse the lists with reciprocal rank fusion (and why you can skip score normalisation)

Reciprocal Rank Fusion (RRF) merges multiple ranked lists without score normalization. Standard min-max normalization requires you to calibrate scores across unpredictable query lengths and corpus changes. If your BM25 query has no strong term matches, min-max normalization forces the top weak hit to score 1.0 and pollutes your downstream context.

RRF skips raw scores entirely and looks only at the position of each document in each list. The score for document d is RRF_Score(d) = sum(1 / (k + rank_i(d))), where rank_i(d) is the 1-based index of document d in list i, and k is a smoothing constant, typically 60 (Cormack et al., 2009). The constant 60 gives high-ranking documents a meaningful boost without letting a document ranked 1st in a single list overwhelm a document ranked 2nd in both lists.

Here is a clean TypeScript implementation for application code:

interface RankedItem {
  id: string;
  score?: number;
}

export function reciprocalRankFusion(
  vectorHits: RankedItem[],
  bm25Hits: RankedItem[],
  k: number = 60
): { id: string; score: number }[] {
  const scores = new Map<string, number>();

  vectorHits.forEach((item, index) => {
    const rank = index + 1;
    const current = scores.get(item.id) || 0;
    scores.set(item.id, current + 1 / (k + rank));
  });

  bm25Hits.forEach((item, index) => {
    const rank = index + 1;
    const current = scores.get(item.id) || 0;
    scores.set(item.id, current + 1 / (k + rank));
  });

  return Array.from(scores.entries())
    .map(([id, score]) => ({ id, score }))
    .sort((a, b) => b.score - a.score);
}

Documents near the top of both lists vault to the front. Exact token matches missing from vector retrieval still land in the top ten because they rank 1st in BM25.

Step 4: Audit where a two-index setup drifts out of sync

The dual-database architecture fixes retrieval accuracy and costs you operational consistency. Running an external vector store next to an external BM25 store puts a distributed systems problem in your core read path. Every document ingestion, update, or deletion has to execute across both systems.

Three failure modes show up in production:

1. Dual-write partial failures: Your application inserts text into your search cluster, but the vector database times out on embedding indexing. Your search cluster now holds documents that don't exist in your vector index. When RRF runs, vector search can't boost the document, and the missing vector makes ranking unstable.

2. Ingestion latency skew: Inverted indexes like OpenSearch commit documents to translogs and refresh searchers periodically (often on 1-second intervals). Dedicated vector stores commit asynchronously via hierarchical navigable small world (HNSW) graph updates that can take seconds or minutes under write pressure. A query fired right after an update searches two different historical states of your corpus.

3. Hard deletions and soft updates: Deleting a document takes two separate API calls. If the vector delete succeeds but the text index delete fails, BM25 returns deleted documents that your application can't hydrate from primary storage. If you run complex agent memory systems, these ghost records contaminate reasoning loops, as covered in our guide to multi-tenant agent memory.

If you keep separate databases, you have to build dead-letter queues, periodic reconciliation crons, and custom compaction routines to hold off data drift.

Step 5: Set up HelixDB with helix init and helix start dev

You can skip the synchronization problem by running BM25 and vector search in a single engine. HelixDB is an open-source graph-vector database built in Rust that combines graph traversal, approximate vector search, and BM25 full-text search. Vectors and text properties live directly on nodes and edges within one transactional store, as detailed in the HelixDB Quickstart.

Install the Helix CLI to manage your local development environment:

curl -fsSL https://install.helix-db.com | bash

Initialize a new project directory and start the development engine:

helix init hybrid-search-demo
cd hybrid-search-demo
helix start dev

The helix start dev command starts a local HelixDB instance running on SlateDB storage. The database listens for native JSON queries sent to POST /v2/query. Graph writes, vector mutations, and BM25 indexes share atomic transactions: writes commit or roll back together, which removes the two-database sync tax completely.

Next, install the native TypeScript client in your project directory according to the HelixDB TypeScript SDK setup guide:

npm install @helix-db/helix-db

You get identical API mechanics across local testing and cloud deployment.

Step 6: Run BM25 and vector search in one engine with the TypeScript SDK

HelixDB lets you query full-text BM25 indexes and native vector properties from application code using JSON queries produced by its native SDKs. There is no separate query language to learn and nothing compiled or pushed ahead of time. The SDK translates queries into JSON, sends them to POST /v2/query, and the database parses and executes them natively.

Here is how to query both BM25 and vector search over nodes in HelixDB using @helix-db/helix-db (v3.0.4):

import { g, VectorDistanceMetric } from "@helix-db/helix-db";

interface SearchParams {
  queryText: string;
  queryVector: number[];
  limit: number;
}

export async function runHybridSearch(params: SearchParams) {
  // 1. Text search using BM25 index on the 'content' property
  const bm25Query = g
    .nodes("Document")
    .search("content", params.queryText)
    .limit(params.limit);

  // 2. Vector search using the native vector index
  const vectorQuery = g
    .nodes("Document")
    .vectorSearch("embedding", params.queryVector, {
      metric: VectorDistanceMetric.Cosine,
      limit: params.limit,
    });

  // Send queries to POST /v2/query via the client instance
  const [textResults, vectorResults] = await Promise.all([
    bm25Query.execute(),
    vectorQuery.execute(),
  ]);

  return {
    bm25Hits: textResults.data,
    vectorHits: vectorResults.data,
  };
}

Because both indexes live in the same storage engine, documents created or updated in HelixDB are immediately visible to both the BM25 searcher and the vector index, with no index skew. The concepts match those in HelixDB's database architecture.

Step 7: Pre-filter vector search to graph-related nodes, then fuse the results

The biggest limitation of running separate vector and text stores is that neither index understands domain relationships. In an enterprise system or an agent memory graph, a document is rarely an isolated chunk. It belongs to a project, is authored by a user, or relates to a specific system component, as discussed in vector search on graph edges.

HelixDB natively supports vector pre-filtering scoped to related entities. Instead of running similarity search over millions of chunks and then post-filtering by tenant or project metadata, HelixDB scopes similarity search directly over a graph-filtered candidate set. Pre-filters can also run before ranking full-text results.

import { g, VectorDistanceMetric } from "@helix-db/helix-db";

export async function searchProjectContext(
  projectId: string,
  queryText: string,
  queryVector: number[]
) {
  // Traverse to documents linked to this project, then run vector search
  const scopedVectorQuery = g
    .node("Project", projectId)
    .out("CONTAINS_DOC")
    .vectorSearch("embedding", queryVector, {
      metric: VectorDistanceMetric.Cosine,
      limit: 20,
    });

  // Traverse to documents linked to this project, then run BM25 search
  const scopedBM25Query = g
    .node("Project", projectId)
    .out("CONTAINS_DOC")
    .search("content", queryText)
    .limit(20);

  const [vectorRes, bm25Res] = await Promise.all([
    scopedVectorQuery.execute(),
    scopedBM25Query.execute(),
  ]);

  return { vectorRes, bm25Res };
}

Your hybrid search runs against relevant structural context and never scans the whole database.

Conclusion

Combining BM25 and vector search stops your RAG pipeline from dropping exact alphanumeric strings, error codes, and low-frequency proper nouns. You can deploy this pattern today by running an inverted index next to a vector store and joining their outputs with reciprocal rank fusion in application code. But when two-database drift, write timeouts, and metadata de-synchronization start breaking your retrieval layer, you need a unified storage engine.

HelixDB combines graph traversal, native vector search, and BM25 text indexes in a single Rust engine. If you're tired of duct-taping two databases together to get reliable hybrid retrieval, star the project on GitHub and run your hybrid queries inside one database.