Multimodal LLMs: How Models That Read Images and Audio Together Are Built

Why it was needed: the limits of text-only models

LLMs (large language models) originally take only text as input. Yet many real-world work materials are not text: photos, drawings, scanned documents, meeting recordings. Early on, such materials were first converted into text with other models. Images were turned into captions (descriptive text), and audio was transcribed with speech recognition (ASR, STT), and that text was then fed to the LLM.

This approach is simple, but information gets lost along the way. If a caption only says 'there is a cup on the desk', the LLM has no way to answer a question about the cup's color or exact position. Converting speech into text also loses tone and intonation. This happens because the conversion step summarizes in advance without knowing what the question will be.

Multimodal LLMs were built to solve this problem: they let the LLM read images and audio directly while keeping as much of the original information as possible. Models that handle images and text together are also called VLMs (vision-language models).

Core idea: turn everything into tokens

A Transformer-based LLM takes a sequence of tokens as input. The idea behind multimodal LLMs is simple. If images and audio are also converted into sequences of token-like vectors and appended next to the text tokens, the LLM can read the whole thing at once in the same way it always has. The token communication study paper also notes that as Transformers have become widespread, tokens are becoming a new basic unit of information processing, and that in multimodal LLMs both visual and textual information are represented and processed as tokens.

As an analogy, the LLM is a capable analyst who only knows Korean, and images and audio are materials in foreign languages. Instead of teaching the analyst foreign languages from scratch, you add a 'specialist reader (encoder)' who reads the materials and an 'interpreter (adapter)' who renders their content in the analyst's language. The analyst places the interpreted content side by side with their own question and makes a judgment.

The trade-off this design makes is clear. Because already well-trained LLMs and image encoders can be reused, there is less need to train a huge new model from scratch. In exchange, the step that 'aligns' how the two models each represented the world comes to determine quality, and, as shown later, if this alignment is imperfect, the problem of 'the information got in but cannot be used' arises.

How it works: encoder → adapter → LLM

The typical processing order has four steps. First, an image is split into small pieces (patches) and passed through an image encoder, while audio is passed through an audio encoder in short segments. The encoder turns each piece into a numeric vector, that is, an embedding. Second, an adapter (also called a projector) moves these vectors into the same space as the LLM's token embeddings. Third, the resulting visual and audio tokens are concatenated with the text tokens into a single sequence. Fourth, the LLM looks at everything together through attention and generates the answer one token at a time.

Training is also usually split into stages. For example, the adapter alone is trained first to get the 'translation' right, and the model is then fine-tuned more broadly on instruction-following dialogue data. The architecture in the token communication study paper is a good example. In this study, an adapter converts transmitted data into visual tokens and inserts them into an intermediate layer of the vision tokenizer, and a two-stage vision-language alignment training is used so that these tokens fit the MLLM. The adapter is first warmed up with a distillation loss and then matched to textual meaning with an alignment loss.

This architecture also puts a burden on the training system. The heterogeneous parallelism study paper points out that as the encoder and the LLM diverge in scale and context windows grow longer, forcing the encoder to follow a single parallelism layout designed for the LLM limits throughput. The mismatch is largest with long contexts: the LLM side needs context parallelism because of the combined multimodal sequence, while encoder inputs have limited length. The study proposed an abstraction that lets the encoder and the LLM use different layouts within a single training graph.

Design choices: train as one body, or connect specialists?

The architecture above joins the encoder and the LLM into a single model and trains the alignment end-to-end. The training-free orchestration study paper argues that this approach has high data and compute costs from combining heterogeneous modalities and is hard to scale, and proposes a different path: connecting existing modality-specific specialist models without additional training.

This framework has three parts. An LLM controller identifies the user's intent and emits explicit control tokens specifying which specialist models to call and in what order; a text-centric cross-modal memory compresses multimodal evidence into structured records so that specialist models do not have to be called repeatedly on every dialogue turn; and a unified interaction layer handles real-time bidirectional streaming and interruption-aware dialogue. The researchers reported strong performance on several multimodal benchmarks with low orchestration overhead and easy replacement of individual components. Because the control tokens are explicit, routing is also easy to audit (verify).

In contrast, the single-body approach lets the LLM see the encoder's fine-grained information directly, so less information is lost to text compression. As a practical rule of thumb: if you need to add several modalities quickly and swap components often, orchestration is the better fit; if subtle cues in an image determine the answer, an integrated model is the better fit.

The range of modalities is also widening. ChemMLLM paper designed five types of tasks spanning text, SMILES strings (which write molecules as text), and images, handling molecular understanding and generation (including image generation) in one model, and reported better performance than general-purpose MLLMs and performance close to specialist models. Cognitive Supersensing paper changes the reasoning method itself. Instead of the existing approach of extending CoT (Chain-of-Thought) in text space, it attaches a head (LVIP) that predicts a sequence of visual latent embeddings to build a reasoning chain from 'mental images', and uses reinforcement learning to optimize textual reasoning paths grounded in those visual latents.

Where things are heading: the gap between 'seeing' and 'using'

A major trend in recent research is evaluation that rigorously examines whether multimodal models actually make proper use of images. BedBench paper tested models on 13 tasks asking about the attributes of 3D objects; on counting by color, humans got 93.3% right, while the best model reached only 7.2%. The internal analysis is even more interesting. An examination of Qwen3.5-27B found that the exact per-color count information could be recovered at the language model's input stage (probe: 65.0%), yet the model's actual answers were almost never correct (3.0%). This means that even when the encoder and adapter pass the information along, the LLM fails to use it in its answers. In a control experiment with the image removed, some 'successes' occurred even without visual evidence.

CulturalBind paper shows the same gap from a different angle. In an evaluation using 3.2 thousand images of 1,000 cultural concepts across 6 domains, models often knew a cultural concept in text but failed to recognize it in images. Textual knowledge and visual knowledge are separated. Knowledge editing could fix individual recognition errors, but those fixes transferred poorly to the other modality, and performance dropped quickly when edits were applied in succession.

Both studies target precisely the weakness of the 'turn everything into tokens' design discussed earlier. Being placed in the same sequence does not mean the knowledge of the two modalities is integrated into one. This is why the BedBench researchers proposed evaluating not only accuracy but also exact correctness, consistent use, and image dependence together.

Limitations and open issues: what to weigh in practice

First, statements without visual evidence, that is, hallucinations. The medical VLM study paper notes that models can produce clinical statements that are unsupported by or contradict the image, and proposed CLEAR, which splits a generated answer into phrases and estimates risk by locally altering the corresponding regions. On MIMIC-CXR, it raised hallucination detection AUPRC from 85.10% to 90.23%. However, this is only detection performance, not actual clinical performance, and in high-risk fields such as medicine, final human review is necessary.

Second, image inputs become a new attack channel. SUA paper showed that sensitive information that machine unlearning has made a model 'forget' may in fact merely be hidden. Adding a single learned noise pattern to input images brought the erased content back, and this noise worked even on previously unseen images. Ownership is another issue. The multimodal fingerprinting study paper points out that because many MLLMs share a language backbone, existing provenance-checking methods are easily confused, and proposed AttnPrint, which uses cross-modal attention distributions as a fingerprint, and DistillTrace, which tests for unauthorized distillation using outputs alone (experiments on 19 architectures and 154 model instances).

There are three common mistakes: believing that high benchmark accuracy means the model is looking at the image to answer, assuming that knowledge the model has in text will also be recognized in images, and treating image inputs as less risky than text inputs. Before adoption, it is best to test directly whether answers still come out without the image, whether the model gives the correct answer consistently every time, and whether it can point to the regions that support its answer.

Further study

To get the architecture down first, read the token communication study paper for the view that 'visual information is tokens too' and for adapter alignment training, and the heterogeneous parallelism study paper for how the size gap between the encoder and the LLM affects the training system. If you are curious about alternatives to integrated training, the training-free orchestration study paper provides a point of comparison.

If you want to know what models actually cannot do, reading BedBench paper and CulturalBind paper together is recommended. Both have experimental designs worth learning from that distinguish 'the information is there' from 'the information is used'. To broaden the range of applications, look at ChemMLLM paper and Cognitive Supersensing paper.

If you plan to run this as a service, it is useful to read the medical VLM hallucination study paper, SUA paper and the multimodal fingerprinting study paper in turn, and lay out the issues of hallucination detection, image-input attacks and model provenance checking.