Vector, Semantic & Hybrid Search

When tokens do not overlap

Keyword (BM25) search matches tokens. Ask it "my card was declined" and it will not find a document titled "payment authorization failed" because the words do not overlap. Vector search fixes this. An embedding model maps text into a dense vector (say 768 or 1536 dimensions) where semantically similar text lands close together. Now "declined card" and "payment authorization failed" are near neighbors even with zero shared words. Retrieval becomes: embed the query, find the nearest document vectors by cosine similarity.

Exhaustively comparing the query to every vector is O(N) and too slow at millions of docs, so you use an approximate nearest neighbor (ANN) index. The two workhorses are HNSW (a navigable small-world graph, excellent recall and latency, high memory) and IVF (cluster the space and search a few clusters, lower memory, tunable recall). ANN trades a little recall for a massive latency win; you tune parameters (efSearch, nprobe) to sit where you want on the recall/latency/memory curve.

Hybrid search: because vectors are bad at exact tokens

Error code E-4021, SKU SKU-99183, version v2.14.0, a person's exact name: these are precisely where semantic similarity fails, because the embedding blurs the exact string. That is why production systems use hybrid search: run BM25 for exact/lexical matching and dense vectors for semantic recall in parallel, then combine.

You cannot just add the scores: BM25 scores are unbounded and dataset-dependent, cosine similarity is bounded 0 to 1, so summing them is meaningless. The clean fix is Reciprocal Rank Fusion (RRF), which ignores raw scores and fuses by rank: each result gets 1 / (k + rank) from each list (k ~ 60) and the sums are combined. A document ranked high by either method surfaces, and the incompatible score scales never touch.

  query --> [ BM25 exact match ]     --> ranked list A
        \-> [ embed -> ANN vectors ] --> ranked list B
                          \-> RRF fuse by rank -> top-k
                                        \-> cross-encoder rerank -> top-n

Retrieve, then rerank

First-stage retrieval (BM25 + ANN) is cheap and optimized for recall: cast a wide net, fetch the top ~100 candidates. Then a cross-encoder reranker (a model that reads the query and each candidate together, far more accurate but far more expensive) reorders just those 100 to produce a precise top 5 to 10. You get the recall of cheap retrieval and the precision of an expensive model, without running the expensive model over the whole corpus.

Interview nuance: two operational realities interviewers probe. Freshness and metadata filtering: you often must restrict to product_id = X or updated_at > T. Prefer pre-filtering (filter the candidate set, then ANN) when the filter is selective, and be aware naive post-filtering can return too few results after ANN. Re-embedding cost: if you change the embedding model, every vector must be recomputed and reindexed, which for hundreds of millions of docs is a real migration, so you version embeddings and roll over like a search alias.

Recap: use embeddings + an ANN index (HNSW/IVF) for semantic recall, run it alongside BM25 for exact tokens like codes and IDs, fuse the two by rank with RRF (never by raw score), add a cross-encoder reranker over the top-k for precision, and plan for metadata filtering and the migration cost of re-embedding.