AI Trend Research

← AI glossary

AI glossary · Basic · Multimodal · Computer vision

VLM

A VLM (Vision-Language Model) is an AI model that can look at images or video and describe their content in text or answer questions about them. It can be seen as a multimodal model that gives a language model 'eyes.'

About this term
Level
Basic · Technical terms you often meet in AI news
Added
2026-10-08
In plain words It is like a team in which a good speaker (the language model) is paired with an interpreter who reads pictures (the vision encoder). The interpreter turns the picture into words the language model understands, and the language model writes an answer based on them. In practice, the two are trained together, so they fit together much more closely than a human team.

How it works

First, the image is split into small patches, and the vision encoder converts each patch into a numeric vector (an embedding). A connector layer (projector) adapts these vectors to the same format as the tokens the language model uses. The language model reads these image tokens together with the user's question text and generates an answer one token at a time. For video, a common approach is to process multiple frames in sequence, like images.

Basic structure of a VLMImage · video → Vision encoder; Vision encoder → Projector; Projector → Language model(Image tokens); Question text → Language model; Language model → AnswerImage · videoVision encoderPatches to vectorsProjectorAdapts to token formatQuestion textLanguage modelReads image and texttogetherAnswerImage tokens
Basic structure of a VLM The image follows the top row and is turned into tokens that enter the language model, where they are read together with the question text to produce an answer.

Why it matters

Much of the information people deal with comes as photos, scanned documents, screens and video. VLMs are used for reading receipts, tables and graphs, describing photos, and computer-use agents that look at a screen to find and click buttons. In robotics, VLMs are evolving into VLAs by adding action output. They matter because they let AI that once handled only text deal with the world as seen through the eyes.

Things to know

They tend to make mistakes reading small text in images, counting objects and judging spatial relationships. Hallucinations also occur, where the model says something is in an image when it is not. They can also be fooled by prompt injection, in which instructions are secretly planted as hidden text inside an image. When using them for important decisions, it is safer to have a person check the results.

Example

Uploading a smartphone photo to a chatbot and asking "Organize this table for me" is a typical use of a VLM. Among the news on this site is a report that Twelve Labs released 'Pegasus 1.6', a VLM that turns first-person video into robot training data; it is an example of applying a VLM's ability to understand video content to robot learning.

Where it came up on this site

Related terms

Multimodal LLM VLA Embedding Token Computer use