Multimodal AI: How Vision-Language Models See and Understand the World

Photo: Google DeepMind / Pexels
Multimodal AI is the family of models that can take in more than one kind of data, usually text plus images, and increasingly audio and video, and reason across them in a single pass. The vision-language model (VLM) is the most mature version: show it a chart and ask what the trend is, and it answers in prose. This article explains how an image becomes something a language model can read, how these systems are trained, what independent benchmarks measure, and why a model that explains a diagram can still miscount circles.
Key takeaways
- One architecture, several inputs: Most multimodal models convert images, audio, and video into sequences of vectors so a Transformer can process them alongside text.
- Three build patterns: Some systems attach a pretrained vision encoder to a language model, some interleave visual information through added attention layers, and some are trained end-to-end across modalities.
- Data decides quality: Datasets range from billions of web-scraped image-text pairs to smaller, curated sets for documents and instructions.
- Perception lags language: On the BLINK benchmark, humans averaged 95.70 percent and GPT-4V reached 51.26 percent.
- Claims need care: Several widely repeated statements about medical, video, and real-time use rest on proof-of-concept studies rather than deployment evidence.
The core idea: turn everything into the same kind of thing
For decades, vision and language were separate research fields with separate tools. A vision system labeled pixels; a language system processed words. A multimodal model collapses the boundary by converting every input type into the same kind of object, a sequence of vectors, so one network can attend over all of it.
An analogy: imagine a translator who only works from written English. To handle a photo, you hire a describer to turn the photo into a precise written account, and the translator works from the account. In a VLM, the describer is a vision encoder that produces numerical summaries of image regions, and the translator is the language model that attends to those summaries as if they were words. The key engineering question is how to make the summaries good enough that nothing the language model needs is lost.
This works because the Transformer, the architecture behind modern language models, does not care what the tokens mean. It only needs a sequence of vectors. The mechanism that lets each element weigh every other element is attention, explained in how transformers pay attention to words. Once image patches are vectors, the same weighing applies across words and pictures at once.
How an image becomes tokens
The standard approach comes from the Vision Transformer (ViT), introduced in a 2020 paper titled An Image is Worth 16x16 Words. Its authors argued that a pure Transformer applied directly to sequences of image patches can perform very well on image classification, competitive with state-of-the-art convolutional networks (the previous standard) while needing substantially less compute to train, when pre-trained on large datasets.
The mechanics are simple. Cut the image into a grid of small squares, flatten each square into a list of numbers, and turn each list into a vector with a learned linear layer. Each vector is one patch token. The Transformer then processes the sequence, so that every patch can attend to every other. For readers who want the older approach, how image recognition works with convolutional neural networks covers it, and computer vision AI: how machines see puts the two in context.
A worked example of the patch arithmetic
The numbers below are an illustration of the arithmetic, not the settings of any specific commercial model.
| Image size | Patch size | Patches per side | Tokens per image |
|---|---|---|---|
| 224 × 224 pixels | 16 × 16 pixels | 14 | 196 |
| 448 × 448 pixels | 16 × 16 pixels | 28 | 784 |
Doubling the image's width and height quadruples the token count, because the area quadruples. Attention compares every token with every other, so its cost grows roughly with the square of the sequence length. A fourfold increase in tokens therefore costs about 16 times as much attention computation, which is why resolution is a real trade-off and why many systems resize or tile images before encoding them.
There is a quieter consequence. A 16 × 16 patch is compressed into one vector. Fine detail inside a patch, such as the exact position of a thin line, must survive that compression to be used later. Large-scale scenes survive well; exact geometry often does not. Keep that in mind when we reach the benchmark results.
Teaching images and words to share a space
A vision encoder trained only to classify objects produces vectors that mean nothing to a language model. The breakthrough that connected the two was learning from captions at scale. In 2021, OpenAI's CLIP was trained on about 400 million image-text pairs gathered from the web, with the pre-training task of predicting which caption goes with which image. The result transferred to more than 30 computer vision datasets, and in a zero-shot setting it matched the original ResNet-50's accuracy on ImageNet without using that model's 1.28 million training examples.
A fitting analogy is a library catalog in which photographs and their descriptions are filed in neighboring drawers. A photo of a dog on a beach lands near the sentence "a dog running on a beach," so a query in either form finds both. That shared arrangement is the embedding space, a concept covered in embeddings explained. A VLM builds on it: the visual representations land close enough to word representations that a language model can use them.
Three ways to build a vision-language model
Systems differ in how they join vision to language. Three patterns cover most of the field.
Pattern one: glue a vision encoder to a language model. A pretrained encoder produces patch vectors, a small trained connector maps them into the language model's embedding space, and the language model reads them as extra tokens. LLaVA is a clear public example. Its authors used text-only GPT-4 to generate instruction-following data pairing language with images, trained a model that connects a vision encoder and a large language model, and reported a relative score of 85.1 percent against GPT-4 on a synthetic multimodal instruction-following dataset. Qwen-VL follows a related plan, adding visual capability to a language model through a visual receptor and a three-stage training pipeline, and its authors released code and models publicly.
Pattern two: interleave images and text with added attention. Flamingo connects strong pretrained vision-only and language-only models and accepts interleaved sequences of images or videos and text. Trained on large web corpora mixing text and images, it could adapt to new tasks from a few examples in the prompt. Its authors reported state-of-the-art few-shot results on many benchmarks, often beating models fine-tuned on far more task-specific data.
Pattern three: train across modalities end-to-end. The GPT-4o System Card describes an autoregressive "omni" model that accepts any mix of text, audio, image, and video inputs and generates text, audio, and image outputs, trained end-to-end across text, vision, and audio, so one network handles all of them. The report gives an audio response time as low as 232 milliseconds, averaging 320 milliseconds.
| Pattern | Typical recipe | Example in the literature | Main trade-off |
|---|---|---|---|
| Encoder plus connector | Frozen or lightly tuned encoder, trained connector, language model | LLaVA, Qwen-VL | Cheap to build; limited by the encoder's detail |
| Interleaved attention | Pretrained vision and language models joined by added layers | Flamingo | Strong few-shot behavior; more complex design |
| End-to-end across modalities | One network trained on all inputs | GPT-4o (per its system card) | Flexible; training details often unpublished |
Public reporting on closed systems is thin. OpenAI's GPT-4 technical report describes image and text input but states it contains no details about the architecture, so assignments of a particular commercial model to a particular pattern beyond what its authors say are guesses.
What the training data looks like
Three kinds of data do most of the work.
Web-scale image-text pairs. The LAION-5B dataset contains 5.85 billion CLIP-filtered image-text pairs, of which 2.32 billion contain English text, and was released as an open resource, along with detection scores for watermarks, unsafe content, and toxicity. Scale helps, but web captions are noisy, often reading like file names or product tags rather than accurate descriptions.
Instruction data. Models that must follow requests need examples of requests and good answers. LLaVA's approach of using a text-only language model to write such examples from image annotations is a cheap way to get volume, with the risk that the generator's errors propagate into the student.
Task-specific document and question data. The DocVQA dataset has about 50,000 questions across more than 12,000 document images, and its authors reported human accuracy of 94.36 percent with a large gap for the baseline models they tested. Datasets like this teach a model to read forms, tables, and scanned pages rather than natural photographs.
The broader training sequence, from pretraining through fine-tuning, is the same one used for text-only models; the Read next list below links to a full walkthrough.
Audio and speech
Audio gets the same treatment. A raw waveform is commonly converted into a spectrogram, an image-like grid showing which frequencies are present at each moment, and then encoded into vectors. Speech recognition at scale is a good illustration of what data volume does: OpenAI's Whisper was trained on 680,000 hours of multilingual and multitask supervised data collected from the web, performed well on standard benchmarks without fine-tuning, and its authors reported accuracy and robustness approaching human levels.
Transcribing words is one task; understanding a conversation is another. Tracking who said what, detecting tone, and inferring what was left unsaid require reasoning that goes beyond converting sounds to text, and published evidence for these abilities is much thinner than for transcription. The GPT-4o System Card describes speech-to-speech interaction as a central focus of its safety work, which is a hint about how new the capability is.
Video and long context
Video multiplies the cost of images. The worked example below again uses an assumed figure of 196 tokens per frame, purely to show the scale.
| Approach | Frames encoded | Tokens at 196 per frame |
|---|---|---|
| Every frame of a 10-second clip at 24 frames per second | 240 | 47,040 |
| One frame per second | 10 | 1,960 |
Sampling cuts the cost by a factor of 24 but discards everything between sampled frames, so a quick event can fall between two samples and be missed entirely. This is the central trade-off in video understanding, and it is why long-context models matter. The Gemini 1.5 report describes models that handle context lengths in the millions of tokens across text, video, and audio, with retrieval above 99 percent up to at least 10 million tokens, compared with 200,000 for Claude 3.0 and 128,000 for GPT-4 Turbo, according to its authors. How a model holds that much material is the subject of context windows explained.
Retrieving a planted fact from a long video or document is a narrower skill than understanding its full story, and published retrieval results should not be read as comprehension results. Video generation, a different problem, is covered in what Sora is.
What these models do well
Reading text inside images is one of the clearest strengths. Qwen-VL's authors report that it can recognize and interpret text in images and locate objects by aligning image-caption-box tuples, and document datasets such as DocVQA exist precisely because reading forms and scanned pages is a distinct, measurable skill. Captioning and visual question answering, the tasks Flamingo was evaluated on, are the other mature categories.
A second strength is flexibility. A single multimodal model can answer a question about a chart, extract a table from a photograph, describe a scene, and then switch to a text-only question. Earlier pipelines needed a separate system for each step, plus glue code. That consolidation is the practical reason VLMs spread so fast, even where each individual skill is no better than a specialist's.
What the benchmarks show about limits
General benchmarks say that VLMs are capable but far from solved. Three independent evaluations are worth knowing.
MMMU. The MMMU benchmark contains 11,500 multimodal questions from college exams, quizzes, and textbooks across six disciplines, 30 subjects, 183 subfields, and 30 image types such as charts, diagrams, maps, and chemical structures. Its authors reported that even GPT-4V and Gemini Ultra reached only 56 percent and 59 percent.
BLINK. BLINK reformats 14 classic computer-vision tasks into 3,807 multiple-choice questions, including relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning. Humans averaged 95.70 percent. GPT-4V reached 51.26 percent and Gemini 45.72 percent, only 13.17 and 7.63 percentage points above random guessing. The authors also found that specialist computer vision models solve these problems much better.
BlindTest. The "Vision language models are blind" study built seven very simple tasks, such as checking whether two circles overlap, counting how many times two lines intersect, identifying which letter in a word is circled, and counting the circles in an Olympic-style logo. Four state-of-the-art VLMs averaged 58.07 percent, with Claude 3.5 Sonnet best at 77.84 percent, while human accuracy was expected to be 100 percent.
| Benchmark | What it tests | Reported result |
|---|---|---|
| MMMU | College-level multimodal reasoning | GPT-4V 56 percent, Gemini Ultra 59 percent |
| BLINK | Core perception (depth, correspondence, multi-view) | Humans 95.70 percent, GPT-4V 51.26 percent, Gemini 45.72 percent |
| BlindTest | Simple geometric and counting tasks | Four VLMs averaged 58.07 percent, best 77.84 percent |
Models tested in these papers are older than the newest releases, and scores may have changed. But the shape of the result has been stable: strong at describing and reading, weaker at exact perception.
Why simple visual tasks are hard
Explaining the gap is partly speculation, so treat the following as reasoning about mechanism rather than a finding from those papers. First, patch compression discards fine geometry, as the earlier arithmetic showed. Second, a language model has a strong prior for plausible text, so when visual evidence is ambiguous it can lean toward the answer that sounds right. Third, training captions describe scenes at a semantic level ("a dog on a beach") and rarely say how many pixels apart two lines are, so the training signal for exact measurement is weak. These are hypotheses consistent with the benchmark pattern, not proven causes.
Hallucinating objects that are not there
A related failure appears in captioning. The paper "Evaluating Object Hallucination in Large Vision-Language Models" reported that such models often describe objects that are not in the image. Objects that appear often in visual instructions, or that tend to appear alongside objects actually in the image, were more likely to be hallucinated. The authors also found that existing evaluations could be skewed by instruction wording and generation style, and proposed a polling-based benchmark called POPE. This is the visual cousin of the text failure described in why AI hallucinates facts.
Medicine and other high-stakes uses
Medical imaging is the application most often cited and most often overstated. Google researchers' Med-PaLM M is a useful reference point because its authors were careful. It handles clinical language, imaging, and genomics with one set of weights, and was evaluated on a 14-task benchmark including mammography, dermatology, and radiology report generation. In a side-by-side ranking of 246 retrospective chest X-rays, clinicians preferred its reports over radiologists' reports in up to 40.50 percent of cases. The authors call it a proof of concept and state that considerable work is needed to validate such models in real-world use. That is a long way from a deployed diagnostic tool.
Regulation moves slowly for the same reason. The US Food and Drug Administration maintains a public list of AI-enabled medical devices, but the page does not state that any listed device uses a large language model or generative AI, and the agency says it will explore methods to identify and tag devices that use foundation models. For background on the narrower, older image-analysis systems used in clinics, see how AI detects patterns in medical scans.
This is general information about the state of research, not guidance on health decisions or on using any tool for diagnosis.
Models that act on screens
Once a model can interpret a screenshot, it can be asked what to click. This is the idea behind computer-use agents: the model sees the screen as an image, proposes an action such as moving the cursor or typing, and a harness executes it. The weaknesses above matter directly here, since mistakes in locating a button or reading small text become wrong actions. What is computer use describes one implementation and its limits.
Real-time interaction is a separate frontier. The latency numbers in the GPT-4o System Card, 232 milliseconds at best and 320 milliseconds on average for audio, are near typical human conversational timing, which is why voice assistants built on end-to-end models feel different from those that chain speech recognition, a text model, and speech synthesis.
Understanding versus generation
Two directions get lumped together. Understanding means image in, text out. Generation means text in, image out. GPT-4o's system card lists image outputs among its capabilities, so a single multimodal model can do both in at least some configurations. Dedicated image generators, though, are usually built differently, most often as diffusion models that start from noise and refine it into a picture under the guidance of a text prompt. How diffusion models work explains that process. The point for this article is that "multimodal" describes inputs and outputs, not a single technique.
Common misconceptions
"Multimodal models see the way humans do." Human vision is active: eyes move, attention shifts, and prior knowledge shapes what is noticed. A VLM typically processes a fixed set of patch tokens from a still image. The BlindTest result, where models struggled with tasks a person finds trivial, is one symptom of the difference.
"Better at language means better at seeing." Language scores and perception scores are measured separately and do not move in lockstep. BLINK's authors found specialist vision models outperform general VLMs on core perception tasks.
"A high exam score means the model can do the job." MMMU measures college-level multimodal questions, which is a proxy. Passing a proxy exam is not the same as performing a radiologist's or analyst's work, and Med-PaLM M's authors explicitly say so for their own results.
"Long context means perfect memory." Retrieval of a planted detail across millions of tokens, as reported for Gemini 1.5, is not the same as integrating a whole video's story. The two are different tasks.
"Open models are far behind." The open Qwen-VL and LLaVA papers reported competitive results for their size at publication, but comparisons change quickly. Read dated benchmark tables instead of assuming a gap in either direction.
What this means
The most important shift is not any single model but the unification: one training recipe, one family of architectures, and one interface that accepts whatever the user has on hand. That is why a photo of a broken appliance, a scanned invoice, and a screenshot of an error message can all be handled by the same tool.
The most important caution is that unification is uneven. Describing, reading, and answering questions about pictures is strong. Exact perception, counting, and long temporal reasoning are weaker, and benchmark evidence for those weaknesses is independent and repeated. A user who knows the shape of the strengths and weaknesses can design around it: trust the model for scene-level understanding, and add a specialist tool or a human check where exact measurement matters.
FAQ
Q: What is the difference between a VLM and a traditional image classifier? A: A classifier maps an image to one label from a fixed list. A VLM accepts an image plus a text instruction and replies in natural language, so it can describe, compare, extract text, or answer follow-up questions. The trade-off is that a specialist classifier trained for one task can still be more accurate on that task, as BLINK's authors found for core perception problems.
Q: Can multimodal models read text inside images? A: Yes, reading text in images is a documented capability. Qwen-VL's authors describe text reading as one of its abilities, and DocVQA, with about 50,000 questions across more than 12,000 document images, exists to measure it. Accuracy varies with image quality and layout, so check outputs on forms and handwriting before relying on them.
Q: How do GPT-4o and Gemini 1.5 differ? A: According to its system card, GPT-4o is a single network trained end-to-end across text, vision, and audio, with low-latency audio replies. According to its report, Gemini 1.5 emphasizes very long context across text, video, and audio, with retrieval above 99 percent up to at least 10 million tokens. They are not strictly comparable, since each report emphasizes different tests.
Q: Are there open-source multimodal models? A: Yes. LLaVA released its data, model, and code publicly, and Qwen-VL released code and models. Flamingo, by contrast, was described in a research paper. Check each project's license before commercial use, since terms differ.
Q: What data trains vision-language models? A: Common sources include web-scale image-text pairs such as LAION-5B (5.85 billion CLIP-filtered pairs), instruction data written partly by other models as in LLaVA, and task datasets such as DocVQA. Quality filtering matters, because noisy captions teach models to describe images loosely.
Q: Why do VLMs fail at counting and simple geometry? A: Independent tests show it happens: on BlindTest, four leading VLMs averaged 58.07 percent on very simple tasks, against an expected 100 percent for humans. Plausible contributing factors include patch compression and the weak training signal for exact measurement, though these are hypotheses rather than established causes.
Sources:
- Dosovitskiy, A., et al. (2020). "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." https://arxiv.org/abs/2010.11929
- "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021. https://arxiv.org/abs/2103.00020
- "Visual Instruction Tuning" (LLaVA), 2023. https://arxiv.org/abs/2304.08485
- "Qwen-VL: A Versatile Vision-Language Model," 2023. https://arxiv.org/abs/2308.12966
- "Flamingo: a Visual Language Model for Few-Shot Learning," 2022. https://arxiv.org/abs/2204.14198
- OpenAI. "GPT-4o System Card," 2024. https://arxiv.org/abs/2410.21276
- "LAION-5B: An open large-scale dataset for training next generation image-text models," 2022. https://arxiv.org/abs/2210.08402
- "DocVQA: A Dataset for VQA on Document Images," 2020. https://arxiv.org/abs/2007.00398
- "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper), 2022. https://arxiv.org/abs/2212.04356
- Gemini Team, Google. "Gemini 1.5," 2024. https://arxiv.org/abs/2403.05530
- "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark," 2023. https://arxiv.org/abs/2311.16502
- "BLINK: Multimodal Large Language Models Can See but Not Perceive," 2024. https://arxiv.org/abs/2404.12390
- "Vision language models are blind," 2024. https://arxiv.org/abs/2407.06581
- "Evaluating Object Hallucination in Large Vision-Language Models," 2023. https://arxiv.org/abs/2305.10355
- "Med-PaLM M" (Towards Generalist Biomedical AI), 2023. https://arxiv.org/abs/2307.14334
- US Food and Drug Administration. "Artificial Intelligence-Enabled Medical Devices." http://www.fda.gov/medical-devices/artificial-intelligence-enabled-medical-devices/list-artificial-intelligence-enabled-medical-devices
Read next:
Follow new articles
Add SciHub101 to your feed reader to see new science and technology stories.
About the Author
SciHub101 is an independent science and technology publication founded and edited by Buğra Sözeri, a software engineer and app developer based in Türkiye. AI tools assist with drafting and research. Source documentation in the existing archive is uneven, and articles are being reviewed and corrected. The Editorial Team byline identifies editorial responsibility; it does not certify specialist or independent human review. Where we compare products, anything built by our own team is labelled as such — see the editorial guidelines and About page for how corrections and disclosures work.
Related Articles

How Image Recognition Works: The Science of Convolutional Neural Networks
In 2010 the best image classifier missed the right label in its top five guesses about 28% of the time. By 2015 that was 3.57%. Here is what changed, and how a CNN actually sees.

AI Reasoning and Chain of Thought: How Models Learn to Think Step by Step
Chain-of-thought prompting and reasoning models have changed what language models can solve. Here is the evidence on why writing out steps helps, how it is trained now, and where the steps stop being trustworthy.

How AI Models Are Trained: The Full Pipeline Explained
Training a large language model is not a single act — it is a five-stage industrial process that begins with petabytes of scraped text, passes through weeks of GPU computation consuming millions of dollars of electricity, and ends with a model that can hold a coherent conversation. Each stage has a specific job, a distinct failure mode, and a measurable cost. Here is how the whole pipeline works, from raw data to deployed product.

What Is Generative AI and How Is It Changing Industries?
Generative AI is the class of machine learning systems that produce new content — text, images, audio, video, code, or molecular structures — rather than classifying or predicting from existing data. It is not a single technology but a family of model architectures sharing one property: trained on enough examples, they can synthesize plausible new outputs in the same domain. Understanding what separates generative from conventional AI explains both why the technology spread so quickly and where its real limits lie.

AI Benchmarks Explained: How We Measure Machine Intelligence
AI benchmarks determine which models get deployed and trusted. Understanding how they work, how scores are computed, and where they fail is essential context for any AI claim.

Advanced Prompt Engineering: Techniques That Actually Work
Beyond 'be clear and specific': the advanced prompt engineering techniques that hold up in published tests, and the popular advice that does not.