AI Trend Research

← AI glossary

AI glossary · Advanced · Classic ML, recsys & search · Language models

BM25

BM25 is a keyword search ranking function that scores each document based on how often, and how rarely, the words in a search query appear in it. It has long been a standard method for search engines and is still used as a core component of RAG.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-08
In plain words It is like a librarian who highlights the words in your question and brings out the books where those words appear most, while ignoring words that appear everywhere, like 'and', and making sure thick books don't win just because they contain more words. Unlike a librarian, though, BM25 does not recognize different words with the same meaning (synonyms).

How it works

For each word in the query, BM25 considers three things and adds up the scores. First, the more often the word appears in a document, the higher the score, but the increase slows as it appears more and more (making it hard to boost rankings by repeating the same word). Second, the fewer documents in the whole collection contain the word, that is, the rarer the word, the greater its weight. Third, it compares document length to the average and adjusts so that long documents don't gain an advantage from word count alone. This calculation runs very fast thanks to a prebuilt inverted index (a list recording which documents each word appears in).

Three factors that determine the BM25 scoreQuery word → Frequency in the document; Query word → Word rarity; Query word → Document length normalization; Frequency in the document → Sum of per-word scores; Word rarity → Sum of per-word scores; Document length normalization → Sum of per-word scores; Sum of per-word scores → Document rankingQuery wordFrequency in thedocumentMore is ↑, withdiminishing gainsWord rarityRarer words ↑Document lengthnormalizationReduces the advantageof long documentsSum of per-wordscoresDocument ranking
Three factors that determine the BM25 score For each word in the query, the three factors are weighed to produce a score; the per-word scores are summed to rank documents.

Why it matters

BM25 can be used right away without any training, is fast, and makes it easy to explain why a document was returned. It is especially strong for searches where exact words matter, such as product codes, people's names, error messages and legal clause numbers. Embedding-based semantic search, which finds results by meaning, tends to miss such distinctive expressions, so today's RAG systems often use hybrid search that combines BM25 and semantic search. It also frequently appears as a baseline when evaluating new search methods.

BM25 and semantic search

BM25Finding by word match

  • Works out of the box, no training needed
  • Strong at finding product codes and names
  • Misses synonyms and different wording

Semantic searchSearch by meaning

  • Requires an embedding model
  • Finds matches by meaning even when wording differs
  • Can be weak at exact matches of distinctive words

공통 The retrieval step in RAG that first finds documents relevant to the question

The two methods are good at finding answers to different kinds of questions, so they are often used together in hybrid search.

Things to know

Because BM25 only looks at literal word matches, searching for 'automobile' can miss documents that only say 'vehicle'. In languages like Korean, where particles attach to words, results vary greatly depending on the quality of preprocessing such as morphological analysis. There are settings that control sensitivity to word frequency and the degree of document length adjustment, so it can be tuned to fit the data. A common setup is to retrieve a generous set of candidates with BM25 in the first stage and then reorder them with a reranker.

Example

Search engines such as Elasticsearch and OpenSearch use BM25 as their default scoring method. When building an internal document chatbot with RAG, a widely used setup sends the question to both BM25 search and vector DB search at the same time, merges the two results, then uses a reranker to select the top documents and pass them to the LLM.

Related terms

Hybrid search Semantic search RAG Reranker Vector DB Embedding