Reranker

How it works

Search is usually split into two stages. In stage 1, a fast method such as vector search narrows a large collection of documents down to tens or hundreds of candidates. Here, the meaning of each document is compressed into a single set of numbers (a vector), and only its distance to the question vector is compared. This is fast, but some information is lost in compression. In stage 2, the reranker reads the question and each candidate document together as a pair, scores 'how relevant it is', and re-sorts them by score. Because it looks at the question and document together, it is more accurate, but it is computationally expensive, so it is applied only to the candidates selected in stage 1, not to all documents.

Why it matters

If a RAG system passes only the top few search results to the AI, a document that is truly needed may be left out because it ranks slightly lower. But passing too many candidates can exceed the amount the AI can read at once (the context window), or the sheer volume of content can cause it to miss important information. A reranker reduces this problem by retrieving a broad set of candidates and then picking out only the few truly useful ones and placing them first. That is why it is often cited as a relatively easy and quick way to improve a RAG system that is not working as well as expected.

How it has evolved

In 2019, Rodrigo Nogueira and Kyunghyun Cho's paper 'Passage Re-ranking with BERT' substantially improved search rankings with a simple approach: feeding the question and a passage together into the language model BERT to score relevance. However, this approach was slow because a large model had to be run for every question-document pair. In 2020, Omar Khattab and Matei Zaharia's 'ColBERT' introduced 'late interaction', which processes questions and documents separately in advance and compares them lightly at the word level at the end, greatly increasing speed while keeping accuracy similar. In 2023, the paper 'Is ChatGPT Good at Search?' showed that a large language model such as ChatGPT can rank documents well given just good instructions, and also proposed a way to transfer that ability to a smaller model (distillation).

Things to know

A reranker only reorders the candidates retrieved by stage-1 search; it cannot find documents that are not among the candidates at all. Also, because it runs the model on each candidate, more candidates mean slower responses and higher costs. In practice, then, it is important to balance how many documents to retrieve in stage 1 and how many to keep after reranking.

Example

Companies that provide search and RAG tools offer rerankers as a separate feature. For example, Cohere describes its Rerank API as taking a question and a list of documents and returning them re-sorted from the most semantically relevant to the question to the least. Vector database company Pinecone also presents 'two-stage retrieval', in which vector search retrieves a broad set of candidates and a reranker then re-ranks them, as a key way to improve RAG performance.

Related terms

RAG Embedding Vector DB Semantic search Hybrid search Context window

References

• Passage Re-ranking with BERT (2019)
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020)
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023)
• Rerankers and Two-Stage Retrieval | Pinecone
• An Overview of Cohere's Rerank Model | Cohere