Embedding

How it works

An embedding model reads input text or images and outputs a list of numbers (a vector) that captures their meaning. During training, it is adjusted so that data with similar meanings get similar numbers and data with different meanings get numbers far apart. So by calculating how closely two vectors point in the same direction (for example, with cosine similarity), you can tell how close their meanings are. For example, "dog food recommendations" and "how to choose food for your pet dog" use different words but end up as nearby vectors.

How it has evolved

At first, "word embeddings," which represent each individual word as a vector, were widely used in natural language processing. In 2019, Reimers and Gurevych's Sentence-BERT modified the language model BERT into a twin (siamese) structure to produce vectors that let whole sentences be compared directly. This made finding similar sentences much faster. In 2020, Karpukhin et al.'s Dense Passage Retrieval (DPR) showed that passages matching a question can be found using embeddings alone instead of keyword matching, and that this works better than traditional keyword search (BM25). In 2022, Muennighoff et al.'s MTEB created a benchmark that compares embedding models across many tasks and languages at once, and confirmed that no single method yet comes out ahead on every task.

Why it matters

Embeddings are the basic ingredient that makes "searching by meaning" possible. They are widely used in semantic search, which finds documents that match in meaning even without the exact search words; in recommendations that show similar products; and in classification that groups similar texts. In RAG in particular, where an LLM looks up external documents and answers with reference to them, the step that turns questions and documents into embeddings to pick out relevant documents is central. Multimodal embeddings that place not only text but also images, audio and video in a single space are now appearing as well.

Things to know

Embeddings are good at capturing "similarity," but they can miss details such as exact numbers, proper nouns and negations ("not ~"). That is why real services often combine them with keyword search in hybrid search, or add a reranker that reorders the results. Also, each embedding model produces vectors in a different form, so switching models means regenerating the stored vectors. Which model works best depends on the task and language, so it is a good idea to test with your own data.

Example

The news covered on this site, "Google releases EmbeddingGemma 2, an open embedding model that puts text, images, audio and video in a single vector space," is a good example. With a model like this, a text search for "sunset at the beach" can also find similar photos and videos. In internal company document search and chatbots, a common approach is to convert documents into embeddings in advance and store them in a vector DB, then, when a user's question comes in, find the closest documents and pass them to an LLM.

Where it came up on this site

• briefing Google releases 'EmbeddingGemma 2', an open embedding model that puts text, images, audio and video in a single vector space 2026-10-07
• briefing Ollama v0.40.0 uses the MLX runtime by default on Apple Silicon 2026-10-07
• Papers ALoDLM: a looped diffusion language model that adjusts loop depth per token 2026-10-07

Related terms

Vector DB Semantic search RAG Hybrid search Reranker Multimodal

References

• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019)
• MTEB: Massive Text Embedding Benchmark (2022)
• Dense Passage Retrieval for Open-Domain Question Answering (2020)
• Word embedding - Wikipedia
• Vector embeddings | OpenAI API