Hybrid BM25 + Vector Search Implementation

Key facts, with sources

• Timescale benchmarks on 50 million 768-dimension embeddings showed PostgreSQL with pgvector and pgvectorscale delivering 28x lower p95 latency and 16x higher query throughput than Pinecone's storage-optimized s1 index at 99% recall. (Tiger Data (Timescale))
• The same benchmark put self-hosted Postgres at roughly $835 per month on AWS EC2 versus $3,241 for Pinecone's s1 tier, about 75% lower monthly cost. (PR Newswire)
• The vector database market was valued around $2.55 billion in 2025 and is forecast to grow at roughly 22% compound annual growth through 2034. (Global Market Insights)
• Pinecone's Dedicated Read Nodes, announced in 2026, claim 77% to 97% cost reduction at scale for sustained high-throughput read workloads compared with standard serverless pricing. (Pinecone)
• Alibaba Cloud's published pgvector HNSW benchmarks document the core tuning trade-off: raising the m, ef_construction, and ef_search parameters increases recall but decreases queries per second. (Alibaba Cloud)

Why neither arm is enough alone

Vector search embeds the query and finds nearest chunks by meaning. It's excellent at 'how do I let users pay later' matching a document about deferred billing — and terrible at 'ERR_CONN_5023', which embeds into a vector near every other error code and nothing in particular. Lexical search (BM25 and Postgres's ts_rank family) is the mirror image: it nails exact tokens — identifiers, product names, legal phrases, code symbols — and completely misses synonyms and paraphrase.

Real query logs are always a mix of both types, often inside a single query ('refund policy for order API v2'). That's why every serious search product — including the managed offerings from Elastic, OpenSearch, Weaviate, and Qdrant — converged on hybrid retrieval as the default. When I benchmarked this in my open-source rag-starter-fastapi project, adding a BM25 arm to pure pgvector retrieval lifted MRR from 0.969 to a perfect 1.000 on the eval set — the queries it fixed were exactly the exact-term ones vectors fumble.

The architecture: two arms, one fusion step

The shape is simple: at query time, run the same user query through both retrievers in parallel, take the top 20–50 from each, and merge the two ranked lists into one. The merge step is where teams overcomplicate things. You cannot naively combine the raw scores — BM25 scores are unbounded while cosine similarities live in [-1, 1], so any weighted sum of raw scores is comparing meters to fahrenheit. The standard fix is to ignore scores entirely and fuse on rank positions, which is what Reciprocal Rank Fusion does.

The lexical arm: Postgres full-text search

If your chunks already live in Postgres (they do in most FastAPI + pgvector stacks), you don't need Elasticsearch. Postgres full-text search with a GIN index gives you BM25-class lexical retrieval — technically ts_rank is not exactly BM25's formula, but for the lexical arm of a hybrid system the difference rarely matters in practice. One column, one index, one query.

-- One-time setup: generated tsvector column + GIN index
ALTER TABLE chunks
  ADD COLUMN search_vec tsvector
  GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;

CREATE INDEX idx_chunks_fts ON chunks USING GIN (search_vec);

-- Lexical arm (query time)
SELECT id, content,
       ts_rank(search_vec, q) AS score
FROM chunks,
     websearch_to_tsquery('english', $1) AS q  -- handles quoted phrases, OR, -
WHERE search_vec @@ q
ORDER BY score DESC
LIMIT 25;

The semantic arm: pgvector

The vector arm is the standard pgvector nearest-neighbor query. Use the same embedding model for documents and queries, and add an HNSW index once the table grows past a few tens of thousands of rows — sequential scan is actually fine below that.

-- One-time setup (pgvector >= 0.5): HNSW index on cosine distance
CREATE INDEX idx_chunks_embedding ON chunks
  USING hnsw (embedding vector_cosine_ops);

-- Semantic arm (query time): $1 is the query embedding
SELECT id, content,
       1 - (embedding <=> $1) AS score
FROM chunks
ORDER BY embedding <=> $1
LIMIT 25;

Fusing the lists: Reciprocal Rank Fusion

RRF assigns each document 1/(k + rank) points per list it appears in, then sorts by total points. Documents that both arms agree on float to the top; documents only one arm found still make the list. The constant k=60 comes from the original RRF paper and works well enough that almost nobody tunes it. This is the entire fusion step:

def reciprocal_rank_fusion(
    rankings: list[list[str]],  # e.g. [lexical_ids, vector_ids], best first
    k: int = 60,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranked_ids in rankings:
        for rank, doc_id in enumerate(ranked_ids, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)


# In the retrieval endpoint:
async def hybrid_search(query: str, top_k: int = 8) -> list[Chunk]:
    lexical_ids, vector_ids = await asyncio.gather(
        lexical_search(query, limit=25),
        vector_search(await embed(query), limit=25),
    )
    fused = reciprocal_rank_fusion([lexical_ids, vector_ids])
    return await load_chunks([doc_id for doc_id, _ in fused[:top_k]])

Tuning and what to add next

Three upgrades, in the order they pay off. First, evaluation before tuning: build a set of real queries with known-correct chunks and track hit rate and MRR — without it you cannot tell whether any change helped. Second, a cross-encoder reranker over the fused top 20 (Cohere Rerank or an open-source model like bge-reranker) buys the biggest quality jump after hybrid itself; it's one API call and reorders the final list by actual relevance. Third, weighting: if your eval shows one arm consistently stronger for your corpus, RRF can take per-list weights, or you can simply retrieve more candidates from the stronger arm.

What usually doesn't pay off: swapping Postgres for a dedicated search engine before you've hit its limits. Postgres hybrid comfortably serves corpora into the millions of chunks; the operational cost of running Elasticsearch next to your database is real and permanent. Move only when you need features Postgres lacks — typo-tolerant autocomplete, complex faceting, true BM25 scoring at massive scale.

Common failure modes

The mistakes I see in hybrid implementations during code audits: fusing raw scores instead of ranks (the meters-vs-fahrenheit problem — results look plausible but one arm silently dominates); running the two arms sequentially instead of in parallel, doubling latency for no reason; embedding the query with a different model or preprocessing than the documents, which quietly degrades the vector arm; forgetting the GIN index so lexical queries sequential-scan and time out at scale; and retrieving too few candidates per arm — with top-5 from each, the arms barely overlap and fusion has nothing to work with. Start at 25 per arm, fuse, then cut to your final top-k.

When to hire senior help

Vector search is easy to start and hard to run well at scale, so bring in senior help when recall problems, filtered-query slowdowns, or index rebuild windows start affecting production, since these usually trace to index and schema decisions made early. An experienced engineer can also prevent the expensive mistake of migrating databases when the real problem is chunking or embedding quality. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships RAG — Vector Search projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Adopting a dedicated vector database before checking whether pgvector on the Postgres already in your stack meets scale needs at a fraction of the operational cost.
• ✕Benchmarking only latency and ignoring recall, then shipping an ANN index tuned so aggressively it silently misses relevant documents.
• ✕Discovering metadata filtering requirements late, since filtered vector search behaves very differently across engines and can collapse recall or throughput.
• ✕Upgrading embedding models without budgeting a full re-embed, or worse, mixing embeddings from different models in the same index.

Frequently asked questions

Sources

• Tiger Data (Timescale)
• PR Newswire
• Global Market Insights
• Pinecone
• Alibaba Cloud