What is a vision-language model (VLM)?
A VLM is an AI model that understands images and text together. It can describe a photo or read and interpret the amounts on an invoice.
What is a VLM?
A vision-language model takes both images and text as input and usually produces text as output. When you photograph a sign and ask "what does this say, translate it into English", or upload a product photo and ask "which model is this?", you are using a VLM.
The image capable versions of current assistants such as GPT-4o, Gemini and Claude belong to this category, and there are open source examples such as LLaVA and Qwen-VL.
How does it work?
A VLM usually has two main parts. A vision encoder cuts the image into small patches and turns each patch into vectors the model can work with. These vectors are placed in the same sequence as the text tokens, and a language model processes them together. That lets the model connect "the chart in the top left corner" with a word in your question.
Where are VLMs used?
- Document reading: extracting data from invoices, receipts, IDs and handwritten forms.
- E-commerce: writing descriptions and tags from product photos, searching by image.
- Customer support: interpreting a photo of a fault sent by a customer.
- Accessibility: describing surroundings and images to blind and low vision users.
- Quality control: spotting defective products on a production line from images.
Limits
VLMs can hallucinate too: they may "see" an object that is not in the photo or misread small text. Counting objects and giving their exact positions are still weak spots. For jobs that need an object cut out of an image precisely at pixel level, specialised models such as SAM are a better fit.