Hybrid search and reranking: retrieve, then reorder

Why it was needed: one retrieval method is not enough

For a long time, document retrieval was mainly handled by lexical methods such as TF-IDF and BM25. These score a document by how often, and how rarely, the words in the question appear in it. They are fast and robust, but as the BEIR paper paper points out, they have a weakness called the 'lexical gap'. If the words used in the question do not appear in a document, they cannot find it even when the meaning is the same, and they also ignore word order.

Neural retrieval emerged to fill this gap. A typical example is semantic search, which converts sentences into numeric vectors called embeddings and finds documents with similar meaning. But when BEIR compared 10 retrieval systems on 18 datasets, dense embedding retrieval often did worse than BM25 in new domains it had not been trained on (zero-shot). No single method won on every dataset either.

So in practice, the two are layered. Hybrid search merges the candidates found separately by keyword search and semantic search to reduce missed documents, and a reranker (a re-ranking model) then reorders the merged candidates more precisely. This article explains why the work is split into these two stages.

Core idea: cast a wide net, then choose carefully

A hiring process is a useful analogy. When tens of thousands of applications come in, interviewers cannot read them all. First, a screening filter broadly shortlists people who meet the requirements (stage 1), and then interviewers look closely at only the few dozen who remain, one at a time (stage 2). Stage 1 is fast but rough; stage 2 is accurate but expensive. Chaining them gives quality close to 'reviewing everyone in depth' at a much lower cost.

Retrieval works the same way. A first-stage retriever quickly pulls hundreds to thousands of candidates out of millions of documents. Hybrid search makes this first-stage net two layers thick. The keyword net catches documents whose words match exactly, and the semantic net catches documents that use different wording but match in meaning.

The second-stage reranker reads the question and each candidate document together and judges again whether 'this document really answers this question'. Passage Re-ranking with BERT paper laid this out as a three-stage pipeline: find about 1,000 candidates with BM25, re-score them with a more computation-heavy method, and pass only the top 10–50 on to the answer-generation stage.

How it works: from question to answer evidence

First, documents are prepared in advance. Long documents are split into pieces of suitable length through chunking (document splitting). Each chunk is indexed twice: once as a list of words (for BM25) and once as an embedding vector (for a vector DB).

Second, when a question comes in, the two retrievers search for candidates at the same time. BM25 uses word matches and embedding search uses vector distance, and each produces its own ranked list. The two scores are on different scales, so they are hard to add directly. A common approach is therefore to merge by 'what rank it had' instead of by score. Reciprocal rank fusion (RRF) gives a larger bonus the higher a document ranks in each list and sums these bonuses. A clinical-record question-answering study paper also merged PubMedBERT dense retrieval and BM25 results with weighted RRF.

Third, the reranker re-scores the merged candidates. The most common form is the cross-encoder. The BERT reranker paper concatenates the question as sentence A and the document as sentence B, feeds them into the model together, and computes the 'probability of relevance' from the leading [CLS] vector. The question is limited to 64 tokens, and the question and document together to 512 tokens. A probability is computed separately for each candidate, and the candidates are re-sorted by that probability.

Fourth, only the top few are placed into the LLM's context window to generate the answer. Ultimately, the purpose of reranking is to put the most useful evidence into a limited number of slots.

How it evolved: accuracy, speed, generalization, and LLMs

BERT reranker 2019 paper: Fine-tuned pretrained BERT as a question-document relevance classifier and took first place on the MS MARCO passage ranking task. It was 27% higher in relative terms than the previous best on MRR@10 (a metric that looks at the position where the correct answer appears), and on the dev set BERT Large scored 36.5 versus 16.7 for BM25. Notably, this was achieved using less than 2% of the full training data. However, it is costly because the whole question and document must be fed into a large model for every candidate.

ColBERT 2020 paper: Tackled this cost problem with 'late interaction'. It encodes the question and document separately with BERT and compares word-level similarity with a lightweight computation at the end. Because document-side representations can be precomputed, less work is left for query time. It achieved accuracy similar to BERT-based models while being two orders of magnitude (around 100×) faster and using four orders of magnitude fewer FLOPs per query. Besides reranking, it can also be used for end-to-end retrieval over large document collections using vector-similarity indexes.

BEIR 2021 paper: Evaluated models without training on 18 public datasets covering 9 tasks (fact checking, citation prediction, biomedical retrieval and others). BM25 was a robust baseline, and reranking and late-interaction models performed best on average but were computationally expensive. Dense and sparse retrieval were efficient but often lagged in generalization. These results support the division of labor of 'cheap first stage, expensive second stage'.

RankGPT 2023 paper: Had generative LLMs such as ChatGPT and GPT-4 rank passages directly. With suitable instructions, they matched or beat the best supervised methods. To check whether the models might have seen the test questions in advance (data contamination), the authors built a new NovelEval set based on recent knowledge. Using 'Permutation Distillation', which transfers ranking ability to a smaller model, a 440M model outperformed a 3B supervised model on BEIR.

Where it is heading: real-world validation and lower cost

First, studies that chain hybrid search and reranking end to end in real settings. A long-term clinical-record question-answering study paper deployed and evaluated, inside a hospital, a RAG pipeline of PubMedBERT dense retrieval + BM25 + weighted RRF + MedCPT cross-encoder reranking. On 1,000 question-answer pairs from 200 bariatric surgery patients, reranking raised the rate at which the correct evidence chunk appeared in the top 10 (Hit@10) from 46.6% to 60.6%, and MRR from 0.2371 to 0.3252. Yet the accuracy of answers generated by Qwen3-8B only rose from 44.8% to 48.6%. In other words, answers do not improve as much as retrieval does.

Second, studies that aim to make expensive rerankers cheaper. RidgeRank paper addressed the slowness of multimodal rerankers that handle document page images. Compressing a reranker lowers its accuracy, so the authors proposed a closed-form fusion formula that recovers the signal missing from the reranker score using the first-stage retriever score. By mixing retrieval and reranking scores rather than discarding either, it effectively applies the hybrid idea to the second stage as well. On 12 datasets from ViDoRe 2 and 3, it kept the NDCG@5 gap to the full cross-encoder within 1.2 percentage points while being up to 48× faster.

Third, attempts to shrink the reranker's input itself. RenderRank paper renders document text as images and feeds them to a vision-language model as visual tokens. Because reranking scores many candidates per question, token savings add up across the number of candidates. On 11 BEIR datasets, it used 16.5–35.5% fewer input tokens while reaching an average NDCG@10 of 55.96, outperforming the evaluated text rerankers under 4B.

Limitations and open issues

The tug-of-war between cost and accuracy remains. In BEIR, the reranking and late-interaction models that were most accurate were also among the most expensive. Increasing the number of candidates lowers the chance of missing the correct answer, but raises reranker call costs accordingly. This is why many recent studies aim for 'similar accuracy at lower cost' rather than large accuracy gains.

A reranker can only choose from the candidates the first stage retrieved. The BERT reranker paper paper also noted that BM25 may never have found some of the correct passages in MS MARCO in the first place. No reranker, however good, can recover documents the first-stage net missed. That is why hybrid search weaves the first-stage net in two layers.

Evaluation itself has pitfalls. BEIR reported that because lexical retrieval was mainly used when building the datasets, relevance labels may be biased toward lexical methods, and that when missing relevant documents in TREC-COVID were annotated manually, the scores of non-lexical methods rose substantially. With LLM rerankers, one must also consider whether the model saw the test data during training, which is why RankGPT built NovelEval separately.

Finally, higher retrieval metrics do not improve final answers by the same amount. The clinical-record study paper showed this gap in numbers. Even when good evidence is retrieved, the LLM may fail to use it properly or may hallucinate, so RAG should be judged on retrieval metrics and answer metrics together.

Further study

If you are new to the topic, start with Passage Re-ranking with BERT paper. It is short and shows, in the simplest form, the three-stage 'first-stage retrieval → reranking → answer generation' structure and how a cross-encoder takes its input.

Next, reading BEIR paper gives a sense of how the five approaches (lexical, sparse, dense, late interaction and reranking) differ, and why BM25 is hard to drop. It is the best source for explaining why hybrid search is used. Then learn the speed-up idea of 'precomputing while preserving accuracy' from ColBERT paper, and the use of LLMs as rerankers plus Distillation from RankGPT paper.

For a practical feel, the clinical-record RAG study paper is a useful case that evaluates the typical combination of BM25 + dense retrieval + RRF + cross-encoder on real data. If you are interested in cost, read RidgeRank paper and RenderRank paper together and compare how one cuts cost through score fusion and the other through input compression.