latent
Teaching a model to read was hard enough. Teaching one to look turned out to be a different category of problem.
For years, computer vision and natural language processing were separate fields with separate architectures, separate benchmarks, separate researchers. CNNs handled images. RNNs and then Transformers handled text. If you needed a system that could do both — caption an image, answer a question about a photo — you bolted two models together with a thin integration layer and prayed.
That era is done. The current generation of multimodal models processes images, text, audio, and video through one architecture with shared representations. The practical effect is that the same chat interface I use to debug Python can now read a handwritten note, interpret a medical scan, extract data from a chart, and describe what's happening in a video clip.
From separate models to unified architectures
How we got here, in three phases.
Pipeline approaches (2015–2019). A CNN encoded the image into a feature vector and a separate language model conditioned on that vector to generate a caption. Show-and-Tell, Bottom-Up Top-Down attention, that whole family. They worked, but the two systems never really shared a representation. The image encoder didn't understand language and the language model didn't understand pixels. They were stitched together at the seam.
Contrastive pre-training (2021). OpenAI's CLIP changed things by training an image encoder and a text encoder jointly on 400 million image-text pairs from the internet. The objective was simple: push matching pairs together in embedding space, pull non-matching pairs apart. After training, "a photo of a dog" and an actual photo of a dog landed near each other in the same vector space. CLIP wasn't generative — it could match and classify but not describe — but it gave the field the visual backbone everything else would build on.
Vision-language models (2023–present). The current approach feeds visual information directly into an LLM as tokens, alongside text tokens. The model isn't "seeing" in any biological sense. It's processing a sequence of embeddings, some derived from text and some from image patches, and reasoning over all of them with the same attention mechanism.
How vision tokens work
The technical core is the conversion of images into token sequences the LLM can process. The pipeline:
Patch embedding. The input image is cut into a grid of patches, usually 14x14 or 16x16 pixels. A 224x224 image becomes 256 patches. A 1024x1024 image becomes more than 4,000. Each patch is projected into the model's embedding dimension through a linear layer or a small neural net.
Visual encoding. The patches pass through a pre-trained vision encoder, often a Vision Transformer trained with CLIP or SigLIP objectives. The encoder already knows about edges, textures, objects, spatial relationships. Its output is a sequence of visual embeddings, one per patch.
Projection into LLM space. The visual embeddings live in a different vector space than the LLM's text embeddings. A projection layer — a linear map, an MLP, or a cross-attention module — moves the visual tokens into the LLM's embedding space.
Interleaving. The projected visual tokens get inserted into the token sequence alongside text tokens. A typical input looks like this:
[text] "Describe this chart:" [vis_1] [vis_2] ... [vis_256] [text] "\nBe specific about the trend."
The LLM's self-attention then processes all of those tokens — text and visual — in a single forward pass. Visual tokens attend to text tokens and vice versa. That's what lets the model ground its language in what it sees.
The key models
A few models have shaped where this is right now.
GPT-4V (2023) was the first widely available model to show strong visual reasoning at scale. It could read handwritten math, identify UI components in screenshots, and extract structured data from messy charts. The "circle the error in this screenshot" demos surprised even the researchers.
Gemini 1.5 (2024) pushed the context window far enough to take hours of video as input. Its native multimodality, meaning it was trained from the start on mixed-modality data instead of having vision bolted on later, gave it stronger cross-modal reasoning.
Claude 3 (2024) showed that multimodal capability and careful reasoning could coexist. It's particularly strong on document understanding, chart interpretation, and spatial reasoning.
Open-source contenders like LLaVA, InternVL, and Qwen-VL proved the architecture isn't proprietary magic. LLaVA in particular showed that fine-tuning an open LLM with a CLIP visual encoder and relatively modest instruction-tuning data could produce surprisingly capable results. I've run LLaVA locally on a single GPU and gotten useful answers out of it.
What this actually unlocks
The applications go well past image captioning.
Document understanding. Feed in a PDF, a scanned invoice, or a handwritten form and the model extracts structured data. This replaces fragile OCR pipelines with a system that understands context — it knows the number next to "Total" is a dollar amount, not a quantity.
Chart and graph interpretation. Bar charts, line graphs, scatter plots, tables. The model reads them and answers analytical questions: "which quarter had the highest growth?", "what's the correlation here?". A static visualization becomes a queryable data source.
Code understanding from screenshots. Paste a screenshot of an error, a UI bug, or a design mockup. The model debugs, describes, or implements what's on screen. This is the part that powers a lot of computer-use agents and developer tools.
Video analysis. With enough context window, the model can take video frames as sequences of images, narrate what happens, identify specific moments, and answer temporal questions like "when does the speaker change topics?".
Spatial reasoning. Given a floor plan, a map, or a diagram, the model reasons about distances, adjacencies, and containment. Still imperfect, but improving fast.
What still breaks
Multimodal models aren't oracles. The failure modes I keep seeing:
Fine-grained counting. Ask the model how many windows are in a building photo and it will frequently miss. Counting small, repeated objects is surprisingly hard because the patch-level representation drops fine spatial detail.
Hallucinated visual details. The model sometimes describes objects or text that aren't in the image. The visual version of textual hallucination, and it's harder to catch because verifying needs me to actually look at the image.
Spatial precision. The model can reason about rough relationships ("the cat is on the table") but struggles with precise measurements or pixel-level localization. Tasks like "draw a bounding box around the defect" still need specialized architectures.
Small text in large images. When an image gets downsampled to the model's working resolution, small text turns into noise at the patch level. High-resolution tiling helps but pays for it in token cost.
What comes next
More modalities, tighter integration, native generation.
Current models mostly consume visual input and produce text output. The next generation will generate images, audio, and video natively, not by calling out to a separate diffusion model but through the same architecture. Gemini's native image generation already points that way.
Audio is on the same path that vision walked. Models that listen to speech, understand tone, and respond with generated voice inside a single architecture are starting to appear. The separate pipeline of speech-to-text, LLM reasoning, and text-to-speech is collapsing into a single forward pass.
This isn't one breakthrough. It's representation learning, scaling, and architectural decisions converging into systems that handle the world more like we do — not one sense at a time, but all of them at once.