When AI sees and speaks

A chart arrives in a screenshot. You want to know which quarter had the highest growth. The tallest bar looks obvious, so this should be an easy question for a model that can see images. Except the tallest bar answers a different question: which quarter had the highest value?

A useful answer requires several things to work together. The model has to read the labels, recover the numbers, understand what you mean by growth, and compare the right quantities. Multimodal models make that conversation possible. Understanding how they connect images and language also explains why a polished answer can conceal a mistake at the very first step.

One chart, two different answers

Consider this invented dataset, used here to make the calculation checkable. These are illustrative values, not results from a model evaluation. Imagine them displayed as a quarterly revenue bar chart:

Q4 has the highest revenue. Q2 has the highest quarter-over-quarter growth, both in dollars and as a percentage: it adds $50,000, or 50%, over Q1. The later increases are $30,000 and $20,000, or 20% and about 11.1%.

Now shrink the chart until the bars remain clear but the labels become difficult to read. You can still describe an upward trend. Recovering the exact values is another task entirely. If a model reads Q2 as $130,000, sound arithmetic would identify Q3 as the largest increase. The calculation would be correct for the wrong input.

That separation matters. We need to distinguish what the model perceived from what it did with that information. To see where the two meet, follow the image into the model.

How pixels become something a language model can use

A language model processes numerical representations called embeddings. To give it visual input, one common design turns an image into a sequence of embeddings too.

A Vision Transformer divides the image into patches and converts each patch into a vector of numbers. Its attention layers let those patches exchange information, so a patch containing part of a bar can be interpreted with surrounding labels and shapes. This patch-based approach is described in the original Vision Transformer paper.

Patch size changes the arithmetic. With 14-by-14-pixel patches, a 224-by-224 image contains 256 patches. With 16-by-16 patches, the same image contains 196. A 1024-by-1024 image divided into 16-by-16 patches produces 4,096. Those are grid calculations, not universal API token counts: resizing, cropping, tiling, and compression can change what ultimately reaches the language model.

The visual encoder's output still needs a connection to the language model. In the original LLaVA architecture, a learned linear projection converts CLIP image features into vectors with the same dimensionality as the language model's word embeddings. Those visual vectors and the user's text form the input sequence. Other designs use connectors such as cross-attention, where language representations consult visual features separately. LLaVA's architecture description makes that distinction explicit.

For a design that inserts visual tokens into the sequence, the input looks schematically like this:

[text] "Describe this chart:" [vis_1] [vis_2] ... [vis_256] [text] "\nBe specific about the trend."

The visual tokens aren't hidden captions. They carry learned image features. During answer generation, the language model uses those features together with the question to predict its response. In a causal decoder, attention follows the permitted sequence order; it isn't an unrestricted conversation where every token can see every other token. The answer is also generated over successive steps, not completed in one forward pass.

This is how the question can direct the model toward the chart's labels and comparisons. It also explains why visual access isn't a guarantee of faithful extraction. The model has a learned representation of the image, and that representation has passed through several transformations before the answer begins.

How the pieces learned to cooperate

Connecting vision and language predates today's chat interfaces. Show and Tell, introduced in 2014, combined a visual network with a recurrent language model to generate captions. Bottom-Up and Top-Down Attention later used salient image regions to support captioning and visual question answering. These were trained systems with meaningful connections between vision and language, not merely two unrelated programs pasted together.

CLIP supplied a different kind of connection. Its 2021 paper trained image and text encoders on 400 million image-text pairs. Matching images and descriptions were encouraged to have compatible representations. A photograph of a dog and the description "a photo of a dog" could then be compared in a shared embedding space. CLIP itself matched and classified; it didn't generate a conversational explanation. The CLIP paper documents both the training setup and the transfer experiments.

LLaVA connected a pretrained visual encoder to a language model and trained it to follow visual instructions. That changes what you can ask of the same image: a caption, a comparison, or a follow-up question. Open projects such as Qwen-VL and InternVL developed further approaches to visual-language alignment, reading, and localization.

Commercial releases brought these interactions to broad audiences. GPT-4V in 2023 accepted image questions; Claude 3 in 2024 supported inputs including documents, charts, and diagrams. These milestones establish what users could do. They don't establish that every model uses the LLaVA architecture, or that one architecture is best at every visual task.

What gets lost before the answer

Return to the revenue chart. Resizing the image can blur a digit before any language reasoning happens. Cropping can preserve a number while removing the axis label that tells us whether it represents dollars or thousands of dollars. Higher-resolution inputs can retain more detail, at a cost in processing and, depending on the service, billable image tokens.

There are other failure modes. A model can invent a visual detail, approximate a count of repeated objects, confuse line styles, or locate an object too imprecisely for the task. OpenAI's vision limitations explicitly identify small text, counting, spatial precision, and incorrect descriptions as concerns. Those failures aren't all explained by patch size; learned perception and reasoning can fail even with a clear image.

For our chart, I would first ask for the extracted quarter-value pairs, then check them against the source before calculating growth. If the underlying spreadsheet or database is available, use its numbers directly. That preserves exact values and units while leaving the model free to explain the result. If the screenshot is all you have, a readable crop and explicit transcription give you something concrete to verify.

The same division of work helps with scanned invoices and handwritten forms. A model can connect a value to a field such as "Total" using both wording and layout. OCR, which converts image text into machine-readable text, remains useful; structured PDF extraction can be preferable when the document already contains text. A vision-language model adds flexible interpretation, but it doesn't make extraction checks obsolete.

Screenshots, video, and spoken questions

Once a model can interpret visual input in context, screenshots become useful working material. An error dialog can be discussed alongside the user's description. A design mockup can guide implementation. A floor plan can support questions about which rooms are adjacent, provided exact dimensions come from readable measurements rather than estimates from pixels. The required precision determines how much verification the task needs.

Video adds a time dimension. Gemini 1.5's technical report described processing long multimodal sequences, including video and audio. A model can use sampled frames to discuss events or find relevant moments. But sampling determines which moments it receives. A brief event between sampled frames can be absent from the input, and a question about when a speaker changes topics also needs the speech content. More context lets the model consider more material; it doesn't restore material the input pipeline omitted.

Speech introduces a similar distinction between preserving information and reducing it. A speech-to-text system gives a language model words, but a plain transcript can omit tone, background sounds, or overlapping speakers. In its 2024 GPT-4o announcement, OpenAI described a model trained across text, vision, and audio within one network, including generated outputs. That is a concrete example of tighter integration. It doesn't mean every multimodal product handles every input or output type, or that generating speech is a single computational step.

The practical opportunity is substantial: ask a question aloud while showing a document, discuss what a video contains, or get an explanation of an unfamiliar interface. Each modality contributes evidence the others may lack. Each also introduces its own opportunities to lose information.

An answer you can check

The screenshot question now has a clear answer. Q2 has the highest growth in our example, and the table shows why. A useful multimodal system should help us get from the image to that explanation while making the extracted values available for inspection.

I would judge the result by the parts the task depends on: did it read the right values and units, apply the intended comparison, and explain the consequence? For a broad description, recognizing the trend may be enough. For a financial calculation, an exact count, or a precise location, the evidence needs a stronger check.

Multimodal models make visual material easier to question and discuss. Their most useful role is often to connect what we can see with what we want to understand. Keep the source data close enough to verify that connection, and a fluent explanation becomes something you can actually use.