Photograph a crumpled receipt and ask a chatbot what it contains: reading and condensing documents is core work for these tools. Photograph a chessboard and ask where the pieces stand: OpenAI’s documentation picks that exact example when it concedes its models handle precise spatial location badly. A multimodal model, plainly, takes in more than text — images, audio, video, files — and answers questions about what it was given. This guide targets people already using ChatGPT, Gemini or Claude, who want to know what to ask of a photo or a recording and when to distrust the reply. The limits come from vendor manuals, franker than the advertising.
- Key Point: A single chat window replaces separate tools for text extraction, transcription and object labelling.
- Key Point: Seeing is not drawing: Anthropic says Claude cannot generate, edit or manipulate images.
- Key Point: Every input costs tokens — 258 for a small Gemini picture, 1,920 for a minute of audio.
- Key Point: Counting, tiny type and left-versus-right questions remain documented weak spots.
Comprehension, generation and the meter
Google’s image-comprehension documentation calls its models multimodal from the outset, able to describe, classify or answer questions about a picture without a system trained for one narrow task. The shift is practical: text extraction, transcription and object tagging once demanded three separate programs; now one file and a plain-language question suffice.
Comprehension is not generation. Anthropic’s vision guide states that Claude interprets images only, with no creating, editing or manipulating them; what a model reads and what it outputs are separate matters, and the second depends on the product. Inputs are metered too: a small Gemini image equals 258 tokens, a minute of audio 1,920 — which is why long video fills the window of material a model can hold, and why an assistant forgets the opening of a long conversation.
Ceilings by vendor, October 2026
Ceilings shift with each release, so read this as developer documentation sampled in October 2026; consumer apps generally allow less. All three take images: OpenAI lists PNG, JPEG, WEBP and non-animated GIF, up to 1,500 images and 512 MB per API request; Google adds HEIC and HEIF and permits 3,600; Anthropic covers JPEG, PNG, GIF and WebP, capped in claude.ai at 20 images per message, 10 MB each and 8,000 by 8,000 pixels.
Gemini’s audio manual is the fullest: 9.5 hours per request, with transcription, translation, timestamps and speaker separation, plus recognition of non-voice sounds such as birds or sirens. Compression to 16 kbps makes the original’s fidelity matter less than expected, though background noise survives. A million-token window handles three hours of low-resolution video or one hour at high, and public YouTube links are accepted; Anthropic’s vision guide has no audio or video entry for Claude.
Day-to-day uses are modest: summarising a photographed document, pulling figures off a receipt, describing a chart, transcribing a recorded meeting, locating the minute where a video raises a topic. Gemini Live can share camera or screen, and its privacy notice says such recordings are not used by default to improve Google’s services.
Counting, fine print and left versus right
Errors cluster where people find things trivial: counting small items, reading minute print, saying what lies left of what. OpenAI lists approximate counts, text needing enlargement, misread rotated or inverted print, trouble with dashed-line and variable-colour charts and with panoramic or fisheye frames, and weaker results on non-Latin scripts such as Japanese or Korean. Anthropic broadly agrees: approximate counts, especially with many small objects; more mistakes on low-quality, rotated or sub-200-pixel images; imprecise coordinates.
Independent work agrees. Vision language models are blind, published in July 2024 and revised in March 2025, set seven tests a person would find easy — whether two circles touch, how often two lines cross. The four leading models of the day averaged 58.07%, the best reaching 77.84%. The visual data was present, the authors found, but the model could not turn it into a correct answer. Manuals still carrying the same warnings in 2026 suggests the problem is unresolved.
Three further cautions: both OpenAI and Anthropic say their models should not interpret specialist scans such as a CT; Anthropic warns Claude cannot tell if an image was AI-generated, and may err if asked; Gemini’s documentation adds a caveat about fast-moving video.
The pattern is consistent, and the makers draw it themselves. These tools are dependable for describing, summarising and transcribing — but fluency is not accuracy. When an answer hinges on a tiny detail, an exact figure or the position of two objects, verify by hand.
Sources & References